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FOREWORD 



Increasingly, 2-year colleges and technical institutes rely 
on periodic evaluation of programs to keep their education offer- 
ings responsive to community needs. With declining enrollments, 
changing technology, and demands for program improvements — all on 
smaller budgets, these schools need to know when to intervene in a 
program for improvement purposes. 

Evaluating and Revising Programs: A Forum of the National 
Postsecondary Alliance reports on program evaluation approaches at 
three member institutions of the National Postsecondary Alliance: 
South Puget Sound Community College, Dallas County Community 
College District, and Triton College. The forum, convened on 
April 16, 1985, brought Alliance members together to share and 
discuss their strategies and concerns about program evaluation. 
This report should be useful to all 2-year college administrators 
and faculty interested .\n alternative approaches to comprehensive 
program evaluation at their own institutions. 

The National Center and the National Postsecondary Alliance 
wish to thank the following forum participants for contributing to 
the report: Luene Corwin, Mercer County Community College; 
Kenneth J. Minnaert, South Puget Sound Community College; Ted 
Martinez, Jr., Dallas County Community College District; and 
Maurice Lemoine, Triton College. James P. Long, Director of the 
Natio:ial Postsecondary Alliance and Senior Research Specialist at 
the National Center, guided the development of the report. 
Constance R. Faddis of the National Center provided substantive 
editing* Margaret Barbee and Monyeene Elliott provided clerical 
support* Copy editing was performed by Judy Balogh of the 
National Center's Editorial Services. 



Robert E. Taylor 
Executive Director 
The National Center for Research 
in Vocational Education 
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EXECUTIVE SUMMARY 



The demand for educational excellence, combined with 
declining enrollments, increasing budgetary concerns, and changing 
technology, have placed added importance on program evaluation 
processes at community, technical, and junior colleges. Two-year 
college administrators must evaluate all programs if they are to 
know how responsive a given program is to community needs and 
whether it should be left as it is, provided more resources to 
improve it, or phased out entirely. However, the process of 
program evaluation takes many different forms, depending on the 
custom at the institution, the priorities of the community it 
serves, the needs of the students, and the participation 
agree.Tients with faculty. 

This report reviews the proc,ram evaluation approacnes 
utilized at three member colleges of the National Postsecondary 
Alliance, a consortium of postsecondary institutions intent on 
achieving excellence in occupational education. The approaches 
were discussed during a forum held in San Diego, California, on 
April 16, 1985. Each speaker discussed the key features of the 
institutional program evaluation process and shared materials with 
the audience. (Copies of these materials appear in the Appendices 
of this publication.) 

Kenneth j. Minnaert reported on program evaluation at South 
Puget Sound Ccnanunity College, an intermediate size school serving 
the Olympia, Washington, area. The college began as a vocational- 
technical institute and became a community college in 1970. The 
evaluation process reviews approximately three programs every 
year, and has eight components: review of a program by full-time 
faculty members, part-time instructors, knowledgeable administra- 
tors, support staff, employers of students formerly enrolled in 
the program, currently enrolled students, former students who have 
been away from the program for awhile, and program advisory com- 
mittee members. 

One evaluation questionnaire asks 19 questions of full-time 
faculty, administrators, and support staff. A separate form is 
used for part-time faculty. Employers receive a form of their 
own, too. Student appraisal is given special attention. In 
addition to the three annual in-depth evaluations of programs, the 
college also performs cursory evaluations of every program it 
offers, in order to meet budgetary and other decision-making 
needs. 

Ted Martinez, Jr., reported on the program evaluation process 
for the Dallas County Community College District, a large multi- 
college entity serving approximately 25,000 occupational students 
annually. Program evaluation on that scale (123 occupational 
programs) is a complex undertaking, particularly since Texas has 
mandated j.ts own vocational education evaluation system (VEES) . 
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Although the Dallas colleges fulfill the state evaluation require- 
ments, administrators found the VEES evaluation process inadequate 
to their needs, and so they developed their own to use in addition 
to the state's* 

The district's evaluation system reviews all similar programs 
offered at the various colleges at the same time. All evaluations 
are conducted separately, with coordination provided by the 
district office. A three-phase process uses carefully worded 
guidelines to assist appropriate administrators, division chair- 
persons, and others to collect needed data from the community, 
students, faculty, and others. Program demand receives particular 
attention. Programs are then prioritized for action at the top 
administrative levels of the district. 

Maurice Lemoine discussed the program evaluation approach of 
Triton College, a large single-campus college serving 25,000 
students in the Chicago suburbs. Beginning in 1984, the Illinois 
Community College Board mandated that all community colleges 
conduct formal review of at least 20 percent of their programs and 
academic support areas each year. Program review focuses on 
student and employer demand, program cost, and program quality. 

Each year Triton College's three-tiered approach reviews 75 
programs with a light "screening" (tier 1), 53 programs with a 
focus on 1 or more of 11 areas (tier 2), and 6 programs with a 
comprehensive review. In this way, Triton is able to conduct 
annually at least a '"snapshot" review of each program it offers, 
using a management-by-exception (i.e., "don't fix it if it"s not 
broken") approach. The evaluation also takes a close look at 20 
percent of the academic support operation areas each year. As a 
result, administrators can discern trends as yearly data are 
accumulated, and can focus on specific needs of those programs 
whose data signal significant problem areas. 
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INTRODUCTION 

by Luene Corwin 

Dean for Academic Affairs, Mercer County Community College 

Trenton, New Jersey 

This forum on evaluating and revising programs at 2-year 
colleges is an activity of the National Postsecondary Alliance. 
The National Postsecondary Alliance is a nationwide consortium of 
community colleges, technical institutes, junior colleges, and 
vocational-technical schools dedicated to improving postsecondary 
occupational education through mutual cooperation. The Alliance 
benefits from the sponsorship of the Kational Center for Research 
in Vocational Education at The Ohio State University. 

The National Postsecondary Alliance concentrates on a number 
of major themes of interest to its members. One theme has been 
high technology. In 1984, members jointly produced a booklet 
called Preparing for High Technology: A Guide for Community 
Colleges. Also they conducted seven National Covxferences on High 
Technology in New Jersey, Illinois, Texas, Florida, California, 
Ohio, and I^orth Carolina. These conferences attracted 800 persons 
from 46 industries and hundreds of 2-year colleges. The Alliance 
has continued its high-tech theme through a variety of hands-on 
workshops to update faculty members in fast-changing occupational 
areas. 

Another major theme of Alliance activity has been economic 
development and the community college. In 1983, Alliance members 
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published a guidebook on the topic, and held three national con- 
ferences in TexaSi^ Ohio, and Washington that attracted nearly 300 
participants. 

The Alliance theme for 1984-85 is Keeping Current in Tech- 
nologyr Marketingr and Financing. A variety of relevant acti- 
vities have enabled Alliance members to assist each other in these 
and other areas, including competency-based education, program 
evaluation, articulation, community needs assessment, use of 
advisory committees, comprehensive institutional planning, program 
phase-out, online electronic communications, and other topics. 
Publications are prepared especially for Alliance members and 
according to their specifications. 

Alliance members meet twice a year. They have a toll-free 
hotline to the National Center, receive regular newsletters, and 
in general help each other avoid "reinventing the wheel." A 
particularly helpful activity has been a sharing of ideas among 
members of the Alliance during the semiannual Alliance meetings. 
During a recent meeting, members exchanged ideas on a variety of 
topics and discovered that many have strong approaches to program 
evaluation. As a result, the current forum has been organized to 
examine program evaluation approaches at three member institu- 
tions: South Puget Sound Community College, Dallas County Com- 
munity College District, and Triton College. 

These three institutions represent quite different 2-year 
postsecondary structures. South Puget Sound Community College in 
Washington State is fairly small and serves a small city and semi- 
rural area. Dallas County Community College District in Texas is 
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a multicampus entity serving a major city and surrounding areas. 
Trit.on College in Illinois serves a large part of the Chicago 
suburbs. 

More and moref program evaluation is becoming an important 
issue for 2-year colleges and technical institutes that must be 
accountable for their use of taxpayers' money. All postsecondary 
educr^tors are becoming more interested in the outcomes of educa- 
tion and in program exit criteria. The three institutions report 
ing on their program evaluation approaches here each have common 
components as well as distinctive features that are of special 
interest to these concerns and others. 
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PROGRAM EVALUATION AT 
SOUTH PUGET SOUND COMMUNITY COLLEGE 

by Kenneth J* Minneart 

President, South Puget Sound Community College 
Olympia, Washington 

South Puget Sound Community College (SPS) is located in 
Olympia, Washington, the state capital. The college district 
serves a population of about 250,000 people. There is another 
community college in the district, plus two 4--year colleges in 
Olympia. The enrollment this year at SPS is just a little under 
1,800 full-time equivalent students, with a head count of about 
3,500 students. 

The college has an interesting history. It began as a voca- 
tional-technical institute (VTI) under the public school system 
and in 1969-70 become a community college under the new state 
community college system. However, it remained single purposed 
for some time, offering only vocational-technical programs. About 
3 years ago, the college was authorized to offer the community 
college transfer program, as well. Last year, in recognition of 
its new roles, it changed its name to South Puget Sound Community 
College. 

The SPS program evaluation process was developed about 4-1/2 
years ago. Initially, the process was designed strictly to 
evaluate vocational-technical programs. The college obtained a 
small summer internship grant for one of the instructors to 
develop this process. SPS has not used this process to evaluate 
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academic programs, but some of its components could apply to 
academic programs, as well. 

SPS has used the process to evaluate about 3 programs every 
year — about 1 per quarter — so that about 12 programs have been 
evaluated so far. The e^'aluation process has undergone some 
refinements and modifications, but overall is working quite well. 



The objectives of the SPS evaluation process are as follows: 

• To determine the adequacy of the education or training 
program in preparing individuals for entry-level employ- 
ment, for advancement, or for further education 

• To improve and revise program components 



• To find ways of making more efficient use of space and 
staff 



# To improve the college's ability to make effective use of 
available community resources 



• To collect and analyze information needed as a resource for 
planning and decision making at the local level 

• To make use of all evaluation data and other information to 
make decisions about program termination, program expan- 
sion, program modification, or other alternatives (e.g., 
leave a program in place because the evaluation finds it 
successful as is) 

The evaluation process can also help assess the current perform- 
ance of formei students, serve as a planning guide, and assist in 



determining whether further evaluation is necessary. 



The evaluation package has eight major components. The full- 
time faculty members assigned to the program to be evaluated 
normally perform the analysis. With part-time instructors the 
process is a little different. We also ask administrators who 
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work with the program and have knowledge of and a relationship 
with it to complete the evaluation package. Other evaluation 
component G or groups include support staff (e.g., instructional 
technicians, instructional aides, or support people who work 
closely with the program) ; employees of students formerly enrolled 
in the program; currently enrolled students; former students who 
have been away from the program for awhile; and, of course, pro- 
gram advisory committee members. SPS uses this process to conduct 
an in-depth evaluation of about three programs per year, and every 
program receives the evaluation at least every 5 years. 

The in-depth evaluation process assesses many factors of a 
program. One factor is the quality of the instruction delivered 
by the faculty members assigned to the program, as perceived by 
other faculty members, administrators, and the support staff. The 
process uses the same form for those three types of individuals, 
and forms are color coded so they can be easily identified and 
sorted. The evaluation questionnaire asks such questions as the 
occupational goals of the program, and then asks the respondent to 
rate those goals as excellent, good, acceptable, poor, don't know, 
or not applicable. The questionnaire asks 19 questions of full- 
time faculty members, administrators, and support staff. 

Part-time faculty are another group asked to participate in 
the in-depth program evaluations. They are asked somewhat differ- 
ent questions, however, because part-time faculty at SPS generally 
are not involved with developing curricula, working with advisory 
committees, aiding in student placement, or other activities 
expected of full-timers. In addition, many part-time instructors 
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are also actively employed in industry, and they have a somewhat 
different perspective to lend to the evaluation process. For 
these reasons, part-time faculty are also asked to respond to some 
questions in writing, such as what they feel are the major 
strengths of the program, major needs for improvement, and the 
like. Finally, because part-timers have different needs for 
institutional support than do full-time instructors, the program 
evaluation questionnaire asks them how the college can help them 
to become more effective instructors. 

College staff work with instructors of targeted programs so 
they know at least several months in advance that their programs 
will be evaluated. In this way, instructors are prepared to and 
staff assist them in reviewing the process and collecting any 
needed data. 

The SPS program e^valuation process also includes an employer 
appraisal form. This form is mailed to the known employers who 
have hired former students of the program of the college within 
the last few years. On the first page of the form, the employers 
basically assess the program from the standpoint of how effective 
the former students are as workers. The second page requests 
information about the employers themselves, as well as information 
about the former students who are now employees (e.g., name of 
employee, how the employee compares with others at the same job 
level, how current the employee's training is relative to industry 
practices and standards, and so forth) . 

Members of the program advisory committee also complete an 
evaluation form. The first page of the form asks the same 
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questions as are asked of employers of former students. The 
second page, however, is different, in that it asks such questions 
as how long the committee member has served on the advisory com- 
mittee, whether the member was informed or educated on his or her 
role as a committee member, how the committee has interacted with 
the program, and the like. These responses help SPS evaluate not 
only the program, but how well it has made use of the program 
advisory committee, how effectively the committee has performed, 
and other crucial data. 

College staff assist program advisory committee members to 
prepare for the evaluation process by making sure they receive 
copies of the institutional commission and goals, the stated goals 
of admission to the program to be evaluated, and so forth. This 
gives the committee members a background and base from which to 
work on their part of the in-depth evaluation. 

Another crucial part of the evaluation process is the student 
appraisal of the program. Both currently enrolled students and 
former students of a program use the same form. The evaluation 
asks for feedback from both of these groups because, in some 
cases, students currently in the program may not be far enough 
into it to make accurate evaluations, just as those who have been 
away from the program for a while may no longer have a fresh 
perspective. 

On an annual basis, SPS performs at least a cursory evalua- 
tion of every one of its programs for budgetary and other 
decision-making needs. Following the close of fall quarter, data 
on fall quarter enrollment are examined to determine student 
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demand for the program. Also examined is the program's cost per 
full-time equivalent student. In other words, SPS compar-^^s the 
costs of the program (the direct instructional costs) with the 
student enrollment. These results are then compared to the state 
of Washington's community college funding model. This model is 
very prescriptive, in that it indicates what amount of funding is 
available from the state for that kind of education/training 
program at a community college. It is important for SPS to know 
how its programs are performing in relation to the state funding 
model and its criteria. 

Community demand is another crucial factor in the annual 
evaluations. SPS examines the placement rates of former students 
via an annual follow-up of each vocational program. Student 
placement data include information on students who have been 
placed directly in the occupations for which they were trained, 
those who have been placed in related occupations, and those who 
are continuing their education. The newest available data on 
employment projections for rhose occupations are also collected. 
Finally, SPS ascertains whether other community colleges or insti- 
tutions in the community offer the same or similar programs. SPS 
uses these and other data to make annual decisions about its 
occupational training programs. 

The SPS evaluation process uses a Wang office automation 
system and an SPS-modified Multiplan package for quick, accurate 
processing and reporting of the evaluation data. A graphics 
package enables the scores of the questionnaires to be displayed 
graphically, the strengths and weaknesses of a program. An 
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historical database on SPS program enrollments allows easy 
analysis of changing enrollment patterns- With this computeri- 
zation, the amount of staff time needed to complete the data 
analysis part of the evaluation is minimal. 

Naturally, the process does not always operate as smoothly as 
hoped. For example, it is sometimes difficult to get persons to 
return the evaluation forms, even though stamped return envelopes 
are enclosed with them. But the computer software has taken most 
of the drudgery out of the analysis process, enabling SPS admin- 
istrators to make program decisions based on reliable, up-to-date 
data. 

As mentioned earlier, the SPS evaluation process has been 
used on quite a few of the programs. Some of these programs have 
been terminated as a result; others have been modified. In many 
cases, the process has provided good documentation to justify 
putting more resources into a program, to acquire more equipment, 
to expand the curriculum or enrollment, to provide inservice 
training for faculty, and so forth. Despite inevitable fears that 
a i:-^^gram will be negatively affected, overall the evaluation 
process has proven to be positive for both programs and faculty. 

SPS would be pleased to share copies of its evaluation forms 
with other members of the Alliance that might be interested in 
instituting a similar program evaluation process. 
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PROGRAM EVALUATION IN A 
MULTICAMPUS COMMUNITY COLLEGE DISTRICT 

by Ted Martinez, Jr, 

District Director for Career and Continuing Education 
Dallas County Community College District 
Dallas, Texas 

The Dallas County Community College District (DCCCD) is 
composed of seven campuses serving approximately 90,000 students. 
About half of these students are enrolled in credit programs and 
half in noncredit programs. The DCCCD has approximately 25,000 
occupational education students in 123 1- and 2-year occupational 
programs. 

As such numbers tend to suggest, program evaluation is not 
easy but is vital in a setting as large and complex as ours. 
Thus, the college district has used an evolutionary process to 
develop its program evaluation procedures. Beginning 5 or 6 years 
ago, the district established a program evaluation planning cycle. 
Since then, the state has mandated a vocational education evalua- 
tion system called VEES. Unf ortunc*tely, the VEES state-mandated 
system does not produce the in-depth information that the DCCCD 
needs in order to evaluate its programs. As a consequence, in 
addition to conducting the required VEES evaluations, DCCCD has 
continued to work on its own evaluation system as the basis for 
its strategic plan for occupational education. 

The function of the DCCCD 's District Office for Career and 
Continuing Education is to coordinate, market, and evaluate the 
occupational and continuina education programs for all of the 
campuses. An assistant director in this office at the district 
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level monitors and coordinates the program evaluation that takes 
place at each campus. In addition, the district office also 
provides helpful information to the campuses, such as cost data, 
enrollment data, and high school student interest data. Much of 
this information is collected by the DCCCD's research office. 

With district staff providing this coordination and support, 
the occupational dean at each campus.- working with the vice- 
president of instruction and the division chairperson for the 
selected programs, has primary responsibility for the evaluation 
of the programs at that campus. In cases where similar programs 
are offered at two or three different campuses (e.g., three air- 
conditioning programs at three different sites) , each is evaluated 
in the same semester. The evaluations are all conducted 
separately, but coordination of information is provided by the 
district office. 

As devised through an intensive yearlong development process, 
the DCCCD's strategic plan for occupational education ^ ^ts forth a 
three-phase program evaluation system. In phase I, the occu- 
pational dean begins by meeting with appropriate campus staff to 
identify and schedule the programs to be evaluated. When the 
schedule is completed, the district office, the occupa-tional 
dean, and the vice-president of instruction appoint an occupa- 
tional education task force and assign responsibilities. Task 
force members represent the campus and include the occupa-tional 
dean, continuing education, occupational and liberal arts faculty, 
division chair, and a business office representative. Program 
advisory committee members are invited as appropriate. The task 
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force begins an intensive data collection process with individual 
action plans designed to ensure accountability and to guide these 
key players through their data-collecting ref:>ponsibilities. The 
collected data becomes the basis for a program profile that is 
compiled by the occupational dean. The profile configures data 
around four categories — capability, funding, importance of pro- 
gram, and market viability — as follows: 

• Capability includes information on instructional factors, 
instructional personnel^ facilities, and equipment. The 
instructional factors include data on curriculum analysis, 
program completers, performance, costs,- and the like. The 
intention is to determine the ability of a program to 
provide appropriate instruction. 

• Funding includes student statistics, the cost per contact 
hour, and comparison data with other programs at DCCCD. 

• Importance includes data on how important that progran. is 
to the campuF, and to the district. 

• Market viability includes employment demand data from the 
state's 1202 Commission, the Texas Employment Coirmission, 
the U.S. Department of Labor, local employer question- 
naires, as well as input from the semiannual meetings of 
the program advisory committee. Student interest data 
collected by the district office are also included. These 
data come from career interest surveys with local Explorer 
Scouts and intermediate school district students. 

Phase II involves occupational program analysis. In essence, 

it is a modification of what is knov/n as a portfolio analysis m 

business. This phase depends on the data that has been compiled 

by the task force through the program profiles. These data are 

plugged into the program analysis process as it addresses its 

service area through consideration of student, general community, 

and business community. The task force uses the data to reach 

consensus (or majority) in each area. 
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Basically, programs offered at a specific campus should 
deliver current and up-to-date instruction to students in its 
service area. Each campus serves a designated sector of the 
county. Thus the program analysis examines external factors such 
as the service area involved and the types of incoming students, 
as well as the needs of the general community. Under general 
community, the process asks, "To what extent is the DCCCD the only 
provider in Dallas County? If we discontinue the program at this 
campus, will there be a negative reaction?" For example, one 
campus, Brookhaven College, recently attempted to discontinue a 
child development lab, but an outcry from the community convinced 
Brookhaven to retain the program. Because service areas for each 
DCCCD campus are very different, each campus is asked to examine 
its services in four areas: instructional services to business 
and industry, involvement with business and industry, high-tech 
emphasis, and entrepreneurship. The program analysis uses 
district wide weighted scores in the student and general community 
columns. Each individual campus determines the weighted score for 
their service area related to the business community. 

The task force then invites the division chair and program 
faculty to participate in the analysis. The task force assigns a 
ranking from 1 to 5. The ranking times the weighted scores 
results in a total score for each program. This process results 
in a prioritization of local programs in descending order. The 
programs are then grouped into three categories (high, medium; or 
low priority) to determine what priority a program has for the 
college service area. 
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In phase III, the task force uses a series of matrices to 
compare service area priority (external factors) to several 
internal factors (capability, funding, importance, and market 
viability). The task force must reach consensus on the relative 
capability, funding, importanctj, centrality, and market viability 
for each program by asrigning a high, medium, or low rating. As 
you might expect, this is not an easy task. A program might rank 
very high on its ability to provide excellent, up-to-date instruc- 
tion, but the need for the program in its local service area may 
rank very low. A robotics program at campus X, for example, may 
be state of the art and well equipped, yet hav(- low demand in a 
service area whose primary economic basis is agriculture. 

Finally, the task force makes recommendations about improv- 
ing, maintaining, or relaxing one -or all of the internal factors. 
These recommendations are forwarded to the vice--president of 
instruction and president for administrative decision making, 
planning, and implementation. 

Currently, the DCCCD is in the process of evaluating all of 
its programs. At the conclusion of this process, program profiles 
will have been completed for all 123 programs in the system. All 
programs at each campus will be ranked according to its score. 
Based on these analyses, the DCCCD will make strategic decisions 
about how to proceed with each program — cutting back or enhancing 
funding, increasing marketability, and so forth. Finally, with 
the program information from each of the campuses, the DCCCD will 
develop a district wide plan for all of our occupational education 
programs. 
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DCCCD is currently tied into a 2-year operational planning 
cycler but there will be cases where evaluations are needed sooner 
than the operation cycle would require. These will be conducted 
on an as-need basis. 

Naturally, the evolving evaluation process has hit a few 
snags. For examplef when the process was first initiated, persons 
assigned to occupational education task forces were somewhat 
intimidated by the program profile used to guide the data collec- 
tion process. But as they became more involvedr this complexity 
began to excite them. The profiles increased their work effec- 
tiveness by helping them examine a program critically yet fairly. 

Another potential problem was that some programs seemed to 
end up at the bottom of the rankings. But even in these cases the 
faculty generally already knew where their programs were weak and 
how they would rank. It is true that through this process a 
program may be phased out. However, the DCCCD has emphasized that 
even when rankings are low the evaluation process can be seen as 
positive — that is, the evaluation may reveal how a program can be 
strengthened rather than eliminated. For example, it may suggest 
that funding should be increased, outdated equipment replaced, 
marketing boosted, and so forth. The evaluation process can help 
the DCCCD to develop a strategy to improve the program. 

The DCCCD hopes to expand its evaluation process to areas 
other than occupational programs. For example, one of the 
campuses is looking into ways to uce the process for its arts and 
sciences programs. Overall, the evaluation has been received 
positively and is functioning efficiently and effectively. 
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THE TRITON COLLEGE PROGRAM EVALUATION MODEL 

by Maurice Lemoine 

Dean, Triton College 
River Grove, Illinois 

Triton College is located in western Cook County and borders 
the west side of Chicago. It is a comprehensive community col- 
lege with a current enrollment for all programs of approximately 
:'5,000 and an FTE of around 9,500. 

Triton has had a formal program review since 1979. The first 
attempt to evaluate programs formally and systematically was not 
successful and the current program review system xS still being 
developed. However, I am inclined v.o believe that our early 
experiences may have been a necessary step in the institutionali- 
zation of program review. 

In 1979, a committee was appointed to develop a program 
review system for the institution. Because of faculty concern 
about how program review data and findings would be used, the 
early system was so comprehensive that it proved to be unwieldy 
and finally unworkable. The process involved filling out innumer- 
able forms, with little staff support for the collection of data. 
Actually, faculty were required to meet with research office staff 
to obtain needed data. 

This procedure was inadequate and was an unnecessary burden 
to program coordinators, and, in a few instances, there was con- 
siderable resistance to the process. The first set of program 
evaluation reports submitted to the academic affairs office varied 
from 13 pages to over 300 pages. Since each report was intended 
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to be reviewed and discussed at several levels, the brief reports 
proved to be much more useful for effecting program change. The 
3C0-page report was a wasted effort inasmuch as it was probably 
never reviewed. This process was very effective, however, in 
that ,it provided evidence that this program review process was not 
effective. 

By the early 1980s, it was obvious that the program review 
system needed revision. At about the same time, the Illinois 
Community College Board (ICCB) mandated that all community col- 
leges establish a formal program review system. Triton now had 
two good reasons to appoint a new committee to modify the present 
system. Our current system is the product of this second effort. 

One of the problems with the initial evaluation process was 
its dependence on a single instrument to be used across the insti- 
tution. The new program evaluation system uses one set of forms 
designed specifically for career programs. Another set of forms 
is used to evaluate academic support programs, student affairs 
programs, the learning resource center, and certain continuing 
education programs (e.g., adult basic education, English as a 
second language, the Employer Development Institute, and so 
forth) . A third packet of forms is used in the arts and sciences 
program. The new planning committee recognized the special needs 
of different programs and also that a single-purpose instrument 
was not appropriate for such a complex process. 

Fortunately, the mandate from the Illinois Community College 
Board (ICCB) was very general and permitted the colleges consider- 
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able latitude in developing their own program review process. It 
mandated that by fiscal year 1984 each community college in 
Illinois would review annually at .least 20 percent of its academic 
programs and academic support services. Prudently, the state did 
not attempt to impose an elaborate system upon the community 
colleges. The state requirements stipulated only that each com- 
munity college develop a program review system that addresses 
three criteria: (1) program need, (2) program cost, and (3) 
program quality. 

A statewide committee appointed by the ICCB and chaired by ar 
ICCB staff member has provided assistance to individual community 
colleges upon request. This process encourages each community 
college to develop — at least initially — a program evaluation 
system to meet its unique needs. Perhaps in the future the state 
board will choose an exemplary model for statewide implementation. 
Currently, each college must submit an annual two-page report to 
the ICCB summarizing the findings and recommendations that 
resulteo from the program evaluation process. 

One of the first decisions the new program evaluation com- 
mittee faced was how to merge fhe state mandate into a program 
review system whose primary purpose would be to address Triton's 
local needs. We were not convinced that it was necessary to 
review all programs with the same frequency, intensity, or 
breadth. From our experience, we knew that some programs might 
require frequent comprehensive evaluation, and others might 
require relatively infrequent overall evaluation and more frequent 
narrowly focused evaluation. 
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The committee decided that it was possible to merge local 
needs with state reporting needs by developing the three-tiered 
program review model. The program screening model is used to 
evc'^'^te 100 percent of Triton's career programs each year. Using 
the data generated by the review process^ Triton identified those 
programs hat additionally require either a focused study or a 
comprehensive formal evaluation. The Triton model . uS adapted 
from one originally developed by Pima College in Arizona. The 
program screening model provides computer-generated information 
based on data collected by the college research office. The 
program data is then disseminated to department chairpersons and 
program coordinators. 

Had the college chosen to follow the state board guidelines 
strictly^ Triton would evaluate only 20 percent of its programs, 
or approximately 15 career programs and 6 or 7 support areas 
annually. With the computerized screening model, the college is 
able to review 100 percent of its programs each year, as well as 
the required 20 percent of academic support programs. It can then 
concentrate evaluation efforts on those programs whose data 
indicate the need for additional attention. If Triton used the 
state comprehensive model, the college would conduct a compre- 
hensive evaluation of all programs every 5 years. By using its 
own computerized screening model, exemplary programs will be 
exempted from an unnecessary comprehensive review and weak or 
marginal programs will be evaluated frequently. 

The program screening model is a good example of "management 
by exception," that is, the college directs its attention to those 
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programs and support services that exhibit evidence of need for 
attention. Inasmuch as comprehensive program evaluations are 
costly in terms of the extensive faculty and staff time they 
require, the program screening model indicates where those 
resources should be allocated for maximum effect. If the college 
were to evaluate only 20 percent of its programs annually as 
mandated by the state, some marginal programs would not receive 
the attention they need in a timely fashion, whereas others would 
be evaluated needlessly. Furthermore, infrequent evaluations do 
not provide the important data that identifies emerging trends 
which might be the basis for critical program decisions. 

At Triton College, 75 different career programs are evaluated 
annually. The screening model provides data relating to each of 
the following: 

• Number of majors in the program 

• Enrollments in courses in the specific program or 
discipline 

• Class size trend 

d Cost of the program 

• Ratio of full-time to part-time faculty 

• Turnover of part-time faculty 

• Number of students successfully completing courses in the 
program or discipline 

© Number of graduates 



• Rate of completion of program 
d Job placement 

• Use of instructional space 

The research office also provides additional data relating to 
other program characteristics, curriculum, and cost and revenue 
analysis. Each year the college prepares a detailed cost analysis 
that includes all revenues and josts that are directly related to 
a particular program. These data facilitate trend analysis for 
particular programs. The accompany data show that, overall, 
Triton's career programs had a $14 surplus per credit hour of 
instruction. That surplus is applied to the cost of support 
services and to institutional overhead. Although the college is 
committed to maintaining a comprehensive curriculum, when the cost 
of particular programs becomes excessive, cost-reducing decisions 
must be made. 

Triton is currently in the process of implementing a new 
management program that will determine the break-even point for 
every course section the college offers. This computer model is 
expected to provide important data for program review. By calcula- 
ting the break-even point for the particular section, the new 
management program will enable Triton chairpersons and admin- 
istrators to manage enrollments more effectively and to make 
better use of instructional facilities. 
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After the college research office prepares the program 
screening model information, the associate vice-president for 
academic affairs who is in charge of the Office of Staff and 
Program Development prepares a two-page report on each career 
program. Each report indicates program strengths and areas of 
concern and includes an overall summary signed by the associate 
vice president. The report then goes to the dean in charge of the 
area who discusses it with the program faculty and affords them 
the opportunity to react to the report. The process has gained 
faculty acceptance and is well on the way to institutionaliza- 
tion . 

This past year, in addition to using the screening model, the 
college research office has also collected some new information on 
a regular basis and provided it to program administrators and 
staff. A one-page summary of program strengths and concerns is 
provided to program staff along with a computer-generated profile 
of all the students in the program, employment information from an 
annual follow-up of career program graduates, information from a 
3-year equipment replacement plan, and the program and course 
objectives. This information provides a broad overview of each 
program. 

Finally, the Office of Staff and Program Development analyzes 

the information collected via the program screening model, rates 

all programs, and makes recommendations for further action. 

Reconimendations include these: 

• Repeat only the screening model next year (automatically 
done for all programs) . 
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• Conduct a focus study of one or more of the 11 dimensions 
included in the screening model. Past experience indicates 
that most programs need focus studies on no more than 2 or 
3 of the 11 dimensions. 

• Conduct a comprehensive review of the entire program 
(includes a major study and components commonly found in 
most program review processes) . 

Since Triton began using program review, it has eliminated 
only one program — a photography program that served hobby and 
recreational interests rather than prepare students for careers in 
photography and related occupations. The college currently offers 
limited instruction in photography through the school of continu- 
ing education. 

Triton's new program review process provides very useful and 
well-targeted information. It has streamlined the program evalu- 
ation system. The college research office provides a report card 
each yeat on the ratings of the screened programs. Overall, the 
new system has become more acceptable to faculty and staff than 
any other model that has been used or proposed. 
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SOUTH PU6ET SOUND COMMUNITY COLLEGE 



PkQGRAM EVALUATION PROCESS 



Applying the philosophy of South Puget Sound Coiiinunity College to provide 
high quality services, it is the policy of the college to conduct a 
locally directed program evaluation process of each instructional 
program. The evaluation process is done cooperatively utilizing the 
knowledge and expertise of instructors, administrators, advisory 
coirmittee members, former and current students, and employers of students. 

The purposes of evaluation include: 

1. Determining the adequacy of the education or training in 
preparing individuals for entry level employment, advancement 
and/or further education. 

2. Improving and revising program components and more efficiently 
utilizing staff and space. 

3. Improving staff competence with regard to evaluating, planning, 
and utilizing available community resources. 

4. Collecting and analyzing of information as a resource to 
planning and decision making at the local level ♦ Based on the 
evaluation, decisions may lead to program termination, 
expansion, modification, or alternatives. 

5. Assessing the current performance of former students as a 
planning guide. 

6. Determining whether further evaluation is necessary. 

The evaluation process involves a formal assessment of all of the major 
components of the instructional program through the use of special rating 
instruments as well as data analysis guides. Formal rating instruments 
are used to survey: 

1. Full-time Faculty Assigned to the Program 

2. Part-time Faculty Assigned to the Program 

3. Administrators 

4. Support Staff 

5. Employees of Students Formerly Enrolled in the Program 

6. Currently Enrolled Students 

7. Former Students 

8. Program Advisory Committee Members 



31 



ERLC 



3p 



A data analysis guide Is utilized to collect and analyze data relative to: 



1 . Student Demand 

* Enrollment history and projections 

2. Cost of Program Operation 

* Cost per full-time equivalent student 

* Comparison of cost to funding model 

* Ratio of cost to funding model 

3. Community Demand 

* Placement of former students 1n related employment 

* Placement of former students In non-related employment 

* Former students continuing their education 

* Employment projections 

4. Duplication 

* Other similar or Identical programs available In the region 
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SOUTH PUGET SOUND COMMUNITY COLLEGE 
Mission Statement 



The mission of South Puget Sound Comrounlty College Is to provide a range 
of quality educational programs for career, personal, and avocatlonal 
needs of the residents of Community College District Twelve. As a 
comprehensive community college, we are comitted to a flexible, 
responsive, educational prograac of general college transfer, 
vocational/technical skills development, developmental education, and 
community service for the individual student that centers on a curriculum 
of knowledge, comprehension, and application* We are committed to 
interrelated career-oriented preparation and a general educational 
experience that will prepare the individual with a foundation of values 
and attitudes for a productive and satisfying life* As a college within 
a multi-college district, w« are coewltted to unity of purpose 1n the 
delivery of services identified by residents of the district* To ensure 
a coiTTOunity based focus, the process for program planning and development 
requires citizen participation: integrated within each program will be an 
emphasis of self-worth, tfie development of a work ethic, the practice of 
safe home and work habits and an understanding of sound economic 
principles. 

The following long-term goals have been established to guide the college 
community in accomplishing its mission: 

1. To provide a variety uf equal opportunities for all students, 
including but not limited to handicapped, disadvantaged, 
minorities, and displaced persons within the college's service 
area. 

2. To maintain, expand, and improve current. program offerings to 
meet the career,/ personal , and avocational needs of the district 
population. 

3. To provide a two-year program of study leading to an Associate 
in Arts degree for students who plan to fulfill the general 
educational requirements for a baccalaureate de^^r^e at a 
four-year college or university. 

4. To provide technical and vocational programs with interrelated 
general education experiences to prepare individuals for a 
satisfying and productive life. 

5. To meet the diverse needs of students by identifying the 
educational needs of present and potential students assisting 
them to explore interests, assess abilities, define personal and 
career goals, and plan programs for goal achievement. 
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To maintain the college as an Integral part of the conrounlty 
through citizen participation and maximize the effectiveness of 
the institution through efficient management of human, fiscal, 
and physical resources. 

To develop a functional operations' plan for a continuing 
endowment foundation to provide supplemental resources for 
building revenues to provide financial support for student 
educational needs not funded through the state allocation model. 
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•Program Evaluation Meioo Co Faculty! 
MEMORANDUM 



TO: 

FROM: Dorna Bullpitt, Associate Dean- of Instruction 
vocational Education 

RE: Program Evaluation 

date: December 10 r 1984 

This year we are conducting a study of your occupational prdgram. The 
purposes are to determine how well it is serving the students and cocr^unity 
and how it can be improved. To do this* you. along with other college 
personnel, selected employers, advisory ccinmittee members, former students and 
present students are being asked to evaluate the Ht^lMl Rssiatant program- 

As part of the evaluation process, please complete the enclosed appraisal form 
and return it to me by '"^iinij/irY 11 . 19fl5. 

You are requested to complete the form individually . Your response will be 
treated as confMential. Faculty responses will be combined into a 
representative composite for the Medical Assistant program. 

Thank you for your assistance. 



DB/Ct 

Attachment 
REF0034V 
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SOUTH PUGFT SOUND 

COMMUSm COLUCZ 



FOPM A-2 



<;taff Aooraisal of 
d'ccuoational Education 



Program^ 



You are: 



Faculty 
Administrator 
Support Staff 



g*ti;«i ifi^i wiirttons : 
Plo»« oi. c-jcn o« tme - '~£XCULC(ir . n.*ri/ 
lonl..mt Item* Jccerdtm. COlll) • »tra..q. ICQ in.r.J 

J^.o..... tu Lcn .tc«. 2sw., POOH - .n*u«.,uau.. lu^r cn.rj 

;"t,«;J" -nen jparopnou. N/A - not .uul .coolc 

1 General Occuoational education Goals Are: 

2. NminmjJ^i.irlnni] Pm.-r- ^-^-h^^ Participation Of: 

1 Adiinnistrators and instn;ctors 

2 Students and foniier students. 

3 -Advisory comnittees 

3. poc.ir. Of The Planninn — ^^^^^ As A Basis For: 



1 Proacam evaluation. ^ _ . - ■ „^-n-.,n, T 

2 ImoK^n^ntiny. di scontu^Tnng or revising programs 

4 P.-nnra.n P.annm., And r».l,.irinn In-— — ^"-^"^ '^^"'"^ 
1 Labor market and employment training needs of the co^ty 

4 Follow-up studies. . 



5. Learner Performance Q hiectives Are: 

; Clearly stated in writing in measurable terms.. 

2 On file for each course. 

3 used 10 help students assess progress...,^ 

4 Consistent w. th job performance requi rements.^ 



6 P roMOtTon Of ()cc uL>aupr-J].Jj"ca.ticin: 

, ,„forn,s the Qubl i c of _progr... goal s and ^n^^^^ 



l:;rs ,rteSu:;%;;d:;ntrrbort i.;portance of Vhe-program 
SuCCCbSfully <jaiMS coiiinunity support. , . 



7. students Oe^irji'M Jo iJi'l'Jli'l UuCupjitj .Hul I'rouroms Are 
1 Cuun^oled on en-Moyment opportunit>es 



3 



T^edtL euu.tabiv in recruitment oncl onro Iment. - 
Jrovuled service, needed such as financial aid and'hild care,_ 
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BEST COPY AVAILABLE 



Instruction Is Adapted To : 
I Recognize students' occupational objectives.. 



10. 



11. 




2 
3 
4 
5 
6 



MP^t individual needs of students. _ 

PmJiie courses at convenient times and locations.. 
Provide for shop and personal safety procedures 
Be realistic in relationship to industry practices.. 
Provide for uparading and retraining as needed. 



9. Related Courses Are : 

t Relevant to the needs of students. ..^«.,»n 

2 SvaiUb e when needed to complement ^""P^^l^J^'^^^JV^^^i^edir 

3 ProMiding communications, r-nhematics ana reading skills needed. 



Rglated Work fOr CTinical) Experience Is; 



1 Available for aU students at convenient times and locations.. 

2 Coardtnated with classroom instruction. ^ 

3 Panned and coordinated with employer supennsion.__ 



Cauoseltno And Guid ance Services: 

1^ a«» arAfl->.T»bTe for aTT studeirts. 

2 
3 
4 
5 



««» available for aTl stuucin.^. — — 

J5 J^oi^d by qualified and interested statt memoer.? 

aS adapted to student career interests and needs. 

Make use of a variety of relevant resources. , 

Help students with personal problems.^ 



12. Special Persons" Serv ices Are: 

1 Readily available to: 

fl) All disadvantaged students. 
(2) All handicapped students 



2 
3 



(3) All displaced homemaker or reentry stuoents.. 
Conducted by qualified staff. 



Coordinated with occupational instruction. 



13. 



£7aluaterthrough monitoring progress of stuaents. 

..V Ri.. And Discrimination Is G iven Emphasis In: 

1 Program enrollments, recruitment and admission policies.. 

2 Course and program content and teaching methods. 

3 Student counseling and guidance. . — 

4 staff recruitment, selection and promotion. 



14. I 



n.truction In Economics And Consume r Fd.T.rinn Is Included To 
Stuoents: 

1 understand their future responsibilities as employees. 

2 Learn to manage time and money. ___ 



I I 



I I 



! I 



! i 



37 



ERIC 



42 BEST COPY AVAIUBLE 



1 S . '^fiirignts Coinoleting A Program; 




1 Are prepared to meet requirements of job for which trai 

2 Are placed in a job related to their training. 

3 Have high standards of work quality.^ , 

4 Have aood work habits and attitudes. _^ 

5 Are able to adjust to the working environment. 



15. Placement Of Occupational Students : 

1 Is cone through a planned, coordinated system. 

7 Identifies employment opportunities for all stuoents. 
3 Instructs students on how to. apply for a job. 



17. Fo^^o^v-uo Studies Are : 

1 Done through a planned, coordinated system.. 

2 Conducted reaularly for program completers.. 

3 Conducted regularly for program leavers. 

4 Current and on file for each program. 

5 Available for instructors' use. 



13. 



The Adninistrative Structure: 



Provides for qualified administrative personnel who are: 

(1) Conwiitted to and knowledgeable about occupational 
Pducation. , -— 

(2) Given time for planning and evaluation. _. 

Provides a clear delineation of responsibility, authority 

and accountabil ity.^_^ z -,^^,n,errarTnn ' 

Encourages coimiunications between statt and aomimstration.. 



19 . The Instructional Staff: 



' Is adequate in number to: . 

(1) He«t individual student learning needs. 

[2] Assist with student advisement^nd^pUcement.. 

(3: ■ ' ^ ' 



Assist with student advisement ana ^ — 

Maintain contact with employers^d potential employers. 



I I 



yocntionally certified., 
auai if ied by: 

(11 Relevant employment experience. __ — 

Appropriate inservice training and/or e^^^'-J^"";— — 
(3) Teaching competence as evidenced by peer, student and/or 



administrative judgment. 
P-ovidrs interestinq and understandable instruction. 
Receivos inservice training opportunities through, 

(1) 

(2) 

Has a 

(1) 
(i) 



.'unds allocaced for 
Time made available 



instructor oarticipation. 

for instructor particip-2tion. 

suDOort staff to provioa: 
Office and clerical assistance. 
Instructional 



assistance for faculty. 
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*1 FunctionTu well naintained. arvd meet safety standards. 

2 Sufficient in supply to meet the needs of the students. 

T Reoresentative of that used in industry. 

4 Flexible and adaptable to changing instructional approacnes. 



22. 



2T. Instructional Materials And Library Resources Are: 



1 Current and relevant to the occupation. , — 

2 Selected to avoid sex bias and discrimination. 

-> Readily available for student use as needed. 

4 Sufficient in quantity for the students enrolled.. 



Representative Advisory Comtiittees : 

1 Have been appointed for the occupations being seryed. 

2 Meet with sufficient regularity to carry out their functions 

3 Provide input in areas such as curriculum planning, evaluatio 
and training standards. , , ■ 



on 



23. 



[he SuQ cet : 
1 



The operating- budget: 

(1) is planned with- instructor input. _ 

(2) Is adeauate for achieving program objectives. 

(3) Is based upon program priorities. 

The capital buaget: 

(1) Is planned with instructor input. 

(2) Supports proaram objectives adeauately.^ — ^ 

3 P^Svides for'scheduled eouipment repair and replacement 

(4) Provides for the refurPishing and modification of fac.Uti 



es, 
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SOUTH PUGET SOUND 



FORM 



coMMUunr college 



Part-Time Faculty Aporaisal 
of Occupational Education 



Program 




foilowtna iccftis acconJino 
to tr.e scale at the rignc 




£XCc:U£»T • nearly Meal 
GOOD • strong, top third 

ACCIPT«aL£ - averace. ni<J3le third 



Resoond :o each iten. using 



tr.e -Jon't know* and ♦*t/A 
racings wnen opprooriace. 



OO.TT XKOTJ • unaole to evaluate 
N/A ** not appMcaole 



How Uell Does The Occupational Education Pr ogram Provide For: 

Labor market and employment training needs of the coinnunity. 

Upgrading and retraining as needed. , _ 

Related work (or clinical) experience for students. 

Counseling and guidance services for students. 

Special services for disadvantaged, handicapped, displaced 

homemaker or reentry students 

Overcoming sex bias and discrimination. ^ 

Students* preparation to meet requirements of job 

for which trained. - 

Placement of occupational students. ^ 

Sufficient equipment and instructional facilities oo 

meet the needs of the students. 



Equicment and instructional facilities representative 



that used in industry. 



Instructional materials and library resources. 
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SOUTH PUGET SOUND 
coMMusny college 



Part-Time Ficu U y Appraisal Proaram 
of Occupational 'Education 



What are the major strengths of the program in which you instruct? 



What are its major needs for improvement? 



How could the college help you become a niore effective instructor? 



4 J. 
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•Program Evaluation Mefoo to Adalniatrators! 

MgMORAHDUM 



to: 



FROM: 



Dorna Bullpitt, Associate Dean of Instruction 
Vocational Education 



RE: 



Program Bvalixation 



date: December 7, 1984 

ve are conducting studies of several occupational prograw this year, the 
purposes are to determine how well the program are serving the students and 
comraunity, and what changes could sake thea aore effective- The study 
includes the input not only fro« ycur but also faculty, current student:st 
former students, advisory coa«ittee aeBbers, e^loyers and support personnel. 

Attached is a copy of the appraisal form that has been designed for use by 

faculty, staff and administrators. 

^s part of the evaluation process, please omiplete the attached 

eor the Mgdlgai Assistant p rogram and return them to me by Jflmmry 11 » l^M . 



DB/ct 

Attachment 
REF0035V 
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•Proqram Evaluation Metoo to Staff! 

M K M O K A M O .0 M 



to: 
FROM: 

RS: 
DATE: 



Doma Bullpitt, Associate Dean of instruction 
Vocational Education 

Program Evaluation 

December 7, 1984 



we are conducting studies ot "Y^f^J^^:^ '^V^t S^g« could uuUce 
determine how well they are the cc«^ty a^«^^ 

them more effective. The study ^i^f^tti^ SSrsTforaer students, current 
representative employers, advisory cc«ittee meiibers, corm. 

students and staff members, 

attached is a copy of the ^^^^^^^^^^f^ors^ ^UhSlgrthere 
only you as support staff ^^^^,J,^J^^^,^^t be able to complete, 
will be many parts of the -valuation ^,1, the Jte^ 

please do those ^»^^*^ij:!^^^j^'^:i:iSe?^ S^or non-applicable. 
j^55isiant program and checlc tne reMoiwo* « 

.our response combined with th^e^f =^^^,^2^^^ S^Ll^™ 

help in providing an overall P^fJ^^J"^^' ^^.fioS and return it to me by 
program. Please complete the attached appraisal torm «» 



DB/Ct 

Attachment 
REF0148V 
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SOUTH PUGET SOUND 

COMMUSrVYCOUSCZ 



FORM A-2 



^ ,t^ff AnoraisaT of 



Program^ 
You are: 



Faculty 
Administrator 

Suppo'^c Staff 



toH»n*in4| uc«rt JCCordiniJ 
to tr»f scjic Ji tnr ngnt. 
Rc^omi.l 10 cjcn Item, usim) 

racings '•oeo juu''O0»'' Jte. 



^jjQIj . vtrwnq, too in»ril 

. not 4OplicJ0U 



\. r.......l 0ccuUJtiona1 ..rdMrarion Goals Are: 

I Clearly stated in wr i 1 1 ng . ^,.,ns - r 7^s"a?r«r"iliTt r7c foTT 

1 rer:ra^^lsl;rpra^;°r,\SS'nc°rro..- anU serv^es.^ 



^,annuui.Occ^^ 

1 Ad>.nnistrators and instructors 

2 Students and foniier students. 

3 Advisory coiunittees. 



gesuU^OlThe.^^ Are Used As A _Basis^ 

1 Proarain evaluation 

r . .Iir^ 



1 Proorain evaluation. rPvi Tinn pronranis 

2 Unoiementmy, discontinuing or revising p g 



TTI^oT^^-d «.ploy..«t tr.intrc needs of t|™|||_^- 
5 3orre"orncr'r4urrt:ent-TSa^l^75^™.,,encac,o.s. 
A Follow-up Studies. — _ 

5 ,.... ..pr Perfonnance Ob jectives Are: 

■ 7-7;;;;:;7IIi;;7;7 writing m measurable terms.. 
7 On file'for each course. rrrr^ ' 

i Succe.slully gums con...unity support. 

Cuun^cled on en-Ploym^nt omjort^^^^ _ 
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10. 



11. 



12. 



Instruction Is Adapted To : 
1 Recoqnize studonts' occupational objectives, 



Meet individual needs of students. 

?ro5ide courses at convenient times and locations.. 
Provide for Shop and personal safety procedures. 
8e realistic in relationship to industry practices, 
Provide for upgrading and retraining as needed. 



9. Rplated Courses Are: 



14, 



I AvanrbJe'^he'n'nelded fo cSX^n t occupat.ona. ^"Strucuo n— 
3 Jroiidilg coinications, mathematics and reading ski. Is needeo, 



Related Work (Or ClinicaU Experience Is: 
1 



13. 



Available for all students at convenient times and locations.. 
Coordinated with classroom instruction.. 



Coordinatea witu Liaasiuui" nuv..- 

punned and coordinated with employer supervision. 



rni.nsplino And Guidance Services: 



1 
2 
3 
4 
5 



Are available for all students. . — ry--— -■ ^ - 

Zl o^ovided by qualified a.d interested ^T^"^'"^' 
Are adapted to student career interests and ne.'ds..._ 

Make use of a variety cf relevant resources. 

Help students with per^oru ! problems. 



<^npcia1 Persons' S e- vices Are : 

1 Readily available to: 

(1) All disadvantaged i\.Jdents 

(2) All handicapped students 



2 
3 
4 



(3) All displaced homen '-.er or reeiiry students.,,. 
Conducted by qualified Scaff. 



Coordinated with occupational ^^5^*^"^ if:---:--!-— 
Evaluated through monitoring progress of ..tuclen-.. 



n^-^^.nn ... .i.. And Discrimination I s Gi v.njElpJllLiji-iil^ 

1 Proqram enrollments, recruitment and admi ssion ool ' cies . 

2 Course and program content and teacninq netho-is. 

■> Student counseling and guidance. 



Staff recruitment, selection and promotion, 



,n.rr.ction In Economi CAJl nd.Cansunier Education 1 s J ncluC:^j:oJlel£ 
SCuaents: 



'l understand their future responsibilities as emoloyees.. 
2 Learn to manage time and money. _ 



I I 
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15. Students Coinoletino A Program : 

1 Are prepared to meet requirements of job for which trained 

2 Are placed in a joo related to their training. . 

3 Have high standards of work quality.^ , — . 

4 Have good work habits and attitudes. , 

5 Aro able to adjust to the working envi ronment. 




IS. Placement Of Occupational Studen ts^: 

1 Is done throuah a planned, coordinated system. 

2 Identifies employment opportunities for all stuaents. 

3 Instructs students on how to apply for a job.^ 



Follow*uD Studies Are : 

1 Done through a planned, coordinated system. 

2 Corioucted regularly for program completers, 

3 Conducted regularly for program leavers. 

4 Current and on file for .each program- 

5 Available for instructors* use. 



13. The A dministrative Structure : 
1 



Provides for quanr->d acii:^i ni strati ve personnel who are: 
(1) Committed to and knowledgeable about occupational 
education. 



euu'-a mil. ■ -- ■ 

(2) Given c-.ne^for planning and evalua:ion, . 

2 Provic^ss a clear delineation of responsibility, authority 
and accountability. - — — ■ 

3 Encourages conwunica 'ions between start and administration.. 

19, The Instructional Staff: 



i adequate in number co: 

M) Meet individual student le^^rmng needs. 

Assi*=^ with student advisement and placement. ^ 

(3) Maintain contact with employers and potential emoioyers, 



Is 
Is 



voccitionally ccrtif ied.___ 

qua! ified by: 

(1) Relevant employment experience. 

2 Appropriate inservice training and/or experience 
(3) Teacning competence as evidenced by peer, student and/or 



I i 



administrative iudgment. 



Provides interesting and understandable instruction. — 
^^e^oivps inservice training opportunities through: 

runos allocated for instructor participation. — 

Has a 

(1) 
(i) 



Time made available for instructor participation, 



support staff to provide: 

Office and clerical ass^s^ance. __ 

Instructional assistance for faculty. 
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22. 
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Equ ipment A nd InrtructjionaLfaciUtifiS 

~r Functional, well maintained and meet safety standards . . 
2 sSff?J?ent in lupply to meet the needs of the students. 

I ?re::i^irtsrad°ajtr.?^ ^t^x ^^-o..^ -pp----- ^ 

ln.;tructional Materials And ■ ii^r-Arv Resources Are: 

1 Current and relevant to the occupation.^ 

2 Selected to avoid sex hias and discrimination. — 

3 Readily available for student use as needed. 
4- Sufficient in quantity for the students enrolTid: 



Representative Advisory Committees : 
1 Have been appointed for the occupations being served. 



2 
3 



and training standards. , — < — — 



23. The Budget: 



1 The operating budget: 

{^) u olanned vTth instructor input. _ 

(2) IS Kate for achieving program objectives. 

(3) Is based upon program priorities. 

2 The capital budget: 

(1) Is planned with instructor input._^^ 

(?) Supports program objectives adequately. - u>.omonf 

3 PrSv?de for^chedaled equipment ^^P^^-^ ■ 

(4) P^Svides for the refurbishing and modification of facilities. 
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.Program Evaluation Letter to Kmployers! 
December 11, 1984 



Dear , 

Ve are conducting a study of our M>><!if?ii ^^^^^f;^n» progran to 
determine hov? well it is 8i»rving the coovunlty and to deteraine what 
changes could make it nore effective « The study ixx:luKles input from 
representative eoployers, advisory conaitte* n«iib«rs» former 
students, current students and staff ambers. 

Will you participate by cczspletlng and returning the enclosed 
appraisal forms consisting of a siurvey font and a page with 
questions relating to you as an eapioyer of a student froa our 
Mer!irai ^€:^^^^ant program. YouT respcnses are a valuable coaponent 
of the study. 

The ratings should be about the eiqployea whose naae is on the 
employer question page. The responses will be treated as 
confidential infonaationr and neither you nor the eaployee will be 
Identified in any way in the final results of the study. 

A stamped, self -addressed envelope is enclosed for your 
convenience. Ve would ajprevtiate your returning the completed 

appraisal form to us Ky .i^nnary 

If you have any questions regarding this process r please do not 
hesitate to contact me. My telephone number is 754-7711, ext. 212. 

Sincerely, 



Dorna Bullpitt, Associate Dean of Instruction 
vocational Education 

DB/ct 

Enclosures 
REF0038V 
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SOUTH PUGET SOUND 

CaMMUNHT COUSCl 



Emo 1 0 ve r's and Advisnrv Contrnttee Memoers' Aooraisal of Cccuoational Education 



Plejsc rjte cjch of the CXCClLC.'lf • ncariv ideal 

GCOO - scrcng. coo criiM 

POOR • in4U»M»ijci* , low»T 



followinq \l.c^\% JCCOrrtinO 
to t"" ^c.^lo 4t :hf» ri'jnt 
Ro^oon«i to c.icn Item. uMnq 



tre vOn t vnoi-»' Jno ."/A 
rjtin,'. wfi^n JOOroor t j to. 



N/A 



How V/e 



1 Does The firr-joational Education Program Provide For: 



Labor market and employment training needs of the community. 
Business/industry trends 



Job performance requirements and employer recommendations_ 
Upgrading and retraining as needed. 



CoriTnunications, mathematics and reading skills needed. 
Related work (or clinical) experience for students. 
Soecial services for disadvantaned, handicapped, displaced 
homemaker or reentry students. ^ 



Overcoming sex bias and disc* imination._ 

Students understanding their future responsibilities 
as employees. ^ 




Students' preparation to meet requirements of job 
for v/hich trained.^ . 



High standards of work Qual i ty. 

Good work habits and attitudes. 

Ability of students to adjust to the working environment. 
Placement of occupational student's. , 



Follow-uD studies of former students.^ 

Equipment and instructional facilities representative 
)f that used in industry. ^ 
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SOUTH PUGET SOUND 

COMMUNITY COLLEGE 

— EMPLOYERS 



Enioloyer's Appraisal of Occupational Education 

name of Firm^ ^Phone 

Address , r . , 

(Street) (CltTJ Tstatel (ZTp) 

Name of Employee Job Title 

How long has this person been an employee? 

Less than 1 year 1 to 2 years * Over 2 years 

Person completing rating Position 



/ias employee's training current with industry practices and standards? Yes 
Comiren ts : 



:ow does this employee compare with others at the same job level? 

Better About the Same ^ Mot as Good^ 



Coninents: 



Do you have any suggestions for imprcvinq the instnictional orogram? Yes No^ 

Comments : — 
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^ Program Evaluation Letter to Advisory Camnictee Members! 
jecember 7, 1984 



Dear 



We are conducting a siirvey of our ^^^.Mr-%\ :^gai<gi-ant program to 
determine how well it is serving the cotasunity and to determine what 
changes could make it more effective, the study includes input from 
representative employers, advisory crmnitten members # former 
students r current students and staff members. 

will you participate by completing and returning the enclosed 
appraisal form consisting of a survey form and a second page with 
questions relating to your role as an advisory committee member. As 
a member of the Medical A5giayar^t program* your responses will be 
valuable to the study. Ve are locking forward to receiving them. 

jr your convenience r a stamped, self- addressed envelope is 
enclosed. We would appreciate your returning the completed 
appraisal form to us by January Ui .1985.^.^ 

Thank you for your cooperation. 

Sincerely, 



Dorna Bullpitt, Associate Dean of Instruction 
Vocational Bducation 

DB/Ct 

Enclosure 
HEF0037V 
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SOUTH PUGET SOUND 
coMstusny collzce 



FomB 



Employer's and Advisory Committee Members* Appraisal of Occupational Education 



flATtr.r, insT;?MCTic::S: 



PUiic rate each of the 
foMo./mci Items accordino 
to scale at the ri'jnt. 
Rcsscnd to each item, using 
tr.c 'Jcn't know" onO .'J/ A 
ra cin;s wnen aoproorute. 



£XC£LLC;iT • nearly ideal 

GOOD - strcnq, too third 

ACCCPTiCLS - dverj^c. njoale tnird 

POQJ? - inacfjudte, \0\4^;r thip-j 

CO.TT <KCU - un^iole to evaluate 

^^/A - not 400 1 icaole 



How Well Does The Occupational Education Program Provide For : 

Labor market and employment training needs of the cornmunity 
Business/industry trends. ^ 



Job performance requireinents and employer recommendations 
Upgrading and retraining as needed. 



Communications, mathematics and reading skills needed. 

Related work (or clinical) experience for students 

Special services for disadvantaged, handicapped, displaced 
homemaker or reentry students*_ ^ . 



Overcoming sex bias and discrimination, 

Students understanding their future responsibilities 
as employees,^ 



Students' preparation to meet requirements of job 
for vyhich trained, , 



High standards of work quality- ^ 

Good work habits and attitudes, , 

Ability of students to adjust to the working environment. 
Placement of occupational students, 



Follow-up studies of former students 

Equipment and instructional facilities representative 
of that used m industry, 
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SOUTH PUGET SOUND 
coM.viu.vnr ccu.£o£ 

— ADVISORY COMMITTEE 

MEMBERS 



'N-„^,ic:ii Committee 
Advisory_Coirmvttee Apprai_sa_i_ 

oTTrociJoat^nai^^ 



HOW long have you been a men,ber of t.js advisory co^^ittee? 

Less than 1 year__ 1 to 2 years__ Over 2 years_ 
.e.e you provided or.entat,on tra,nin, as to your role as advisory con.,ttee «er^ 



Yes f^o. 



tional pronrani(<-.) m US field. U.e., curr i.u 



the occupational eaucation prooram^s) in its .leirt. 



w.a. ,0 .ou coas<.er as the »aJor s.r=n,«s of t.e occupaUona, ecucaticn pro=ra™(si 
your advisory coimii ttee serves? 



. rhP nnior needs for iniproveii.ent in the occupationa. ecuca 

\j:i.u jo you consider as the major neeri^ " 
tion pro<jrom(s) your advisory conmittee serves. 
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!Program Evaluation Letter to Current Students' 
December 7, 1984 



South Puget sound Co«mity college is conducting a study of several 

^"SeteJSSe S!^"!? the^I^uaticn "e 

to determine how well the prograu are serving not only you as a 
student but also the cownity and b«« the progran can be 
Improved. To do this selected «q>loyers, advisory coanittee 
members, faculty aesbers, foraer students and present students are 
being asked to evaluate the progrMs. w «b 

Will you please participate by coe^leting and returning the enclosed 
appraisal font for the Median »*«HntflnT progra*. Please do not 
f !? !5f f?""* responses will not be identified with you 

individually. Your participation along with that of other students 
currently enrolled wiU be of value in detenHning how well the 
Medical teal a tant progra* is aeetlng your needs and expectations. 

Thank you for your assistance in returning the ccvpleted fora to ae 
by Jaqy^ry 4r 1995 using the enclosed, salf-addressed envelope. 

Sincerely, 



Dcma Bullpitt, Associate Dean of Instruction • 
Vocational Education 



DB/ct 

Enclosure 
REF0156V 
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iProgram Evaluation Letter to Former Students! 
December 14, 1984 



Dear # 

South Puget Sound Coannmlty College is conducting a stud/ of several 
occupational prograas this year to assess how well they prepared you 
for the world of worlc or ftirther education and to determine what 
changes can make thas nore effect lve« Would ycu assist by 
ccispleting the enclosed appraisal font. 

Your responses coyibined with those of other students and former 
students will be of value to us in appraising the Medical Assistant 
program in which you were enrolled* Your responses %#111 be treated 
as confidential and you will not be individually identified in any 
way* 

A self^addressed stamped envelope is mclosed for your convenience. 
Vould you please cocplete and return the evaluation form by J^nuuy 
4, 1?85. 

Ve hope all is going well with you. Please keep us informed of your 
activities and of any way in which wc may be of service to you. 

Sincerely, 



Dorna Bullpitt, Associate Dean of Instruction 
vocational Education 



DB/ct 

Enclosure 
RSF0128V 
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SOUTH PUGET SOUND 

COMMUHrVf COU£CZ 



F0I?1C 



student Ao prai saT of 
Ot'cuoa tiona rEducatfon 



Program 

You are a: 



Current Student^ 
Former Student^ 



Pltfjsc rjce eacn of tne CXClltCnr 

foMowtng ue*n5 occonUno GOOD 

to me scale at tr^c rtgnc. accSPTABIE 

Rescona to ejcn ucm, using POOP 

tre 'Jon't know" ana N/A OO.TT JCMOVJ 

ratings v»nen approprtace. ?l/A 



• nt4r)y ideal 

- jtrcng, too tntrti 

- averace. mdrtle tftird 

- )n4cecu4te« lo>»*fr tntrd 

• unaole to ev4iu4te 

• not 4Ppl tcaole 




S fjdents gesirino To EnroTI In QccupationaT Proorams Are: 



Counseled on employment opportumtiies . 
Treated equitably in recruitment and enrollment. 



Provided serv 



ices needed such as financial aid -and cm Id care. 



The Instructional Program is Ad apted To: 

Recoanize students' occupational objectives 
Meet" individual needs of students 



orovide courses at convenient times and locations.. 
Provide for shop and personal safety proced-jres. 
3e realistic in relationship to industry practices. 



Related Courses Are : 

Relevant to the .needs of students. _ '„c»>...^rin n 

Available when needed to complement ^^^^P^^^^"^ '^""^^^^^^'e^id" 
Providing cotimuni cations, n.athematics and reading skills needed. 

Related '..'ork (Or Clini raU Experience Is: 

Available for all students at convenient times and locations. 



Coordinated with classroom instruction 
Planned and coordinated with employer supervision.. 



rr,..r><;p]ir>a And Guidance Services: 



Are available for all students. _-— 

ire provided by qualified and interested staff memoers.. 

Are adapted to student career interests and needs. 

Make use of a variety of relevant resources. 

Help students with personal problems. _ 



* 

rn r.nnnnir. And consum e r Edu cation Is Included To Hel c 



"'understand their future responsibilities as employses.. 
Learn to manage tine and money. 
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^fudents Coir nletina A Program: 

Are prepared to meet requirements of job for which trained. 
Are placed in a joo related to their trammg.^ 

Placement Of Qccuoational Students: 



r5 done through a planned, coordinated system. . 
Identifies Sloyment opportunities for a students.. 
Instructs students on how to apply for a job. 

rhe Instructional Staff: 



Is adequate in number to: 

-Meet individual student learning needs.. 



fl<;<;i<;t with Student advisement and placement. — 

:f,a?ntai: cSntact with employers and potential employers.. 



Is qualified by relevant employment experience. 
Jrovides interesting and uMderstandable instruction. 

r n.unment And In <^rr..ctional Facilities Used Are: 



Sufficient in suppiy <-u mecL l..^ ' - 

:n.rr..ct1onal Ma ^.'^-^.l. And Library Resources Are: 



Current and relevant to the occupation. 

Selected to avoid sex bias and discrimination.,,. 
Readily available for student use as needed.^^ 
Suff aent in quantity for the students enroTTid7 



1 1 
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erIc ^2 



PAGE 1 



SOUTH PUGET SOU»jO COHNUNITY (XRIEGE 

Inslruction Program Pl^snning CcnniittiM? 
(sup|M>rlin9 dita- \W\tl} 



STUDENT 
OEMAM) 

HU FTE 
1Sd4 


COST 
PER 
FIE 


COST 

fundioQ 
nodel 


lof 

Ratio 


£a|>1o: 
Related 


emu 

DEIV 
Placenent 
wont 

Non- 
Related 


IITY 

1982/83 

Cont. 
their 
education 


Nijii>er 
contacted 


DUPLICATION: 
Uashinaton Ccmunity 

Colleges 
(SouttMest Region) 
(Ft. SteilacocM, Oark, 
Olynpic. Iwr. Colunt>ia, 
Tacona. Centralia, 
Grays Harbor! 



ACCOUNTING 
(2101) 




















AHTHROPOiOGY 
(2115) 


















• 


ART 

(2209) 




















AUTOrOTIVE TECHNOiOGY 
(2401) 




















BIOLOGY 
(2302) 




















8USINESS 
(2114) 




















CKENISIRY 
(2303) 




















CIVIL ENGINEERING 
TECH^^IOGY (2305) 





















00 
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EXPLANATION FOR GSNr/RAL PURPOSE SURVEY 



• EXIT 

Allows operator to exit from the program. 

2. HELP INFORMATION 

Provides operator with information vrtien a problem arises in utilizing the 
program. 

3. CEEATB/CHANOB (Keyword file) 
Allows operator to create fiie by: 

a. Entering number of survey questions 

b. Entering Icey word statement Eor each question 

4. ENTER SURVEY DAtA 

Opertor enters data for questions from each surveys form. - 

5. VIEW SURVEY RESULTS 

Allows operator to view survey results on the screen before printing- 
Shows statements and responses with weighted averages given for each 
response. Autoaatically excludes N/A*s or blanks from averaging process, 

6. VIEW CORRELATIONS 

Allows operator to enter items where correlation is desired, view data and 
request a printout of correlated items. 

7> COPY KEYWORDS 

Saves rekeying keyv^ords when same survey questions are to be asked of a 
new group. 

8. DELETE 

Allows operator to delete data from the survey or to delete the entire 
survey . 

The information compiled from the survey data can also Interface with a 
business graphics software package for the purpsoe of visual charting. 
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** GENERAL PURPOSE SURVEY ** 
* MASTER MENU * 

1 - EXIT 

2 - HELP INFORMATION 

3 - CREATE /CHANGE KEYWORD FILE 

4 - ENTER SURVEY DATA 

5 - VIEW SURVEY RESULTS 

6 - VIEW CORRELATIONS 

7 - COPY KEYWORDS FROM EXISTING SURVEY 

8 - DELETE CURRENT SURVEY (CAUTION) 
ENTER OPTION: 

•Fl' TO ESCAPE 
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Appendix B: 



Dallas County Community College District 
Program Evaluation Guidelines 
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OCCUPATIONAL EDUCATION PROGRAM EVALUATION 



FORMAT GUIDELINES 



Purpose of Evaluation 

In addition to the authority and encouragement stated in the Texas 
State Plan for Vocational Education, the evaluation must be linked 
back to the real purposes and objectives of the college and that of 
occupational education in the district as stated in the DCCCO policy 
manual . 

"The colleges shall monitor the technical and occupational 
training needs of the Dallas area and the communities they 
serve, and shall develop and offer courses which are de- 
signed to equip students, through one and two-year credit 
programs, with the skills and technical knowledge required 
for successful employment in semi-professional or other 
occupational fields." 



Philosophy of Evaluation 

The comprehensive community college, charged with the responsibility 
of meeting a broad spectrum of educational needs, as well as meeting 
manpower demands, must systematically assess its programs' relevancy 
and quality in meeting those needs. The improvement of instruction, 
the updating of programs, and the efficient use of resources coordinat- 
ed with meeting student and community needs are the real purposes of 
evaluation. 



General Procedure 

The following occupational program evaluation format is intended to 
be a minimum-standard guide in evaluating all DCCCD occupational 
programs on a three-year cycle. Campuses are encouraged when and 
where appropriate to go beyond this format. For example, there may 
be unique programs which necessitate the use of a third party outside 
evaluator or a TEA consultation. 

In order for the evaluation results to be timely and useful, every 
effort should be made to complete a program evaluation during a 
12-month period of time. It is recommended that all evaluations 
begin in the Fall. 

Occupational Program Evaluation 1s basically a campus activity. The 
Division Chairperson and the Tech/Occ Dean have active roles in the 
entire process. The Tech/Occ Dean provides over-all guidance and 
supervision of the process on campus. To assist campus personnel, the 
District staff will provide cost, enrollment, student interest, and 
follow-up data. The Assistant Director of Occupational Education will 
serve as a general district coordinator/facilitator for the evaluation 
activity, and will be available to assist in data analysis, formulation 
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Occupational Education Program 
Evaluation Format Guidelines 



of reconmendations, and the preparation of reports. 

The attached flow chart identifies key elements in the evaluation process, 
lists titles of responsible persons, and suggests a time frame. Deviations 
from this format and time frame are permissible. Hovfever, Tech/Occ Deans 
should consult with the District Occupational Education office prior to 
any major deviation. 

The first step in the three-year cycle is for the Tech/Occ Dean and other 
campus administrators to identify the programs to be evaluated each 
. semester. Programs should be identified at least onn year In advance 
of the evaluation process. (A program evaluation schedule is attached.) 
Similar programs should be evaluated during tiie same semester. For 
example, if three campuses have an auto mechanics program, those programs 
.hould be evaluated during the same semester. 

It has been considered advantageous to have an Occupational Program 
Evaluation Comnittee on each campus. The functions of that committee 
include assisting the Tech/Occ Dean in making assignments and sharing 
the work load. It is suggested that the committee member's responsi- 
bilities be planned well in advance of the evaluation process in order 
for those responsibilities to be recorded on the Individual Action Plans 
for each person participating. 

When the data collection and compilation phase has been completed, the 
Tech/Occ Dean shall invite the Assistant Director of Occupational Educa- 
tion to review the data, suggest further collection or analysis, and 
to assist in the formulation of recoirmendations. 

The Division Chairperson shall develop a preliminary draft of the final 
report to be submitted to the Tech/Occ Dean, the Vice President of 
Instruction, and the college President. It Is suggested that the Vice 
President of Instruction convene a program evaluation review (Fleeting 
for all key persons involved or affected to discuss the preliminary 
draft and to determine future action. Upon completicn of the campus 
review, a final report will be developed by the Division Chairperson 
and the Tech/Occ Dean. (The final report is discussed further on 
page three of these guidelines.) 

It is suggested that the final report be kept on campus and a synthesized 
version prepared for submission to the District Director of Occcupational 
Education. 
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OCCUPATIOKAl PROGRAM EVALUATION 
aOW CHART 



District Offlcf «ni Supply: 

- TtxSIS Follow-up 0*U 

- Cost OiU 

• Enrol Intnt Oita 

-High School Stud«nt 
Inttrtst D«U 



Identify and Schtdule 
Prograa EviluitlOfi 



Evftlufttlon Comrlttee 
Hwtinq and Assl^raients 



Individual (s) Tlioe 
Responsible Schedule 

1-3 yrs. 
In advance 



T/0 Dean 



T/0 Dean 



No -ater 
than 2nd week 
of sanester 



I Record , V.P.I. Early Fall 

I Responsibilities , T/D Dean Semster 

i In I Div. Chair 

» I. A. P. « 

L.. .J 



Program 




Student 




Oenand 




Interest 





Instntctlonal 
Factx>rs 



T 



OMplle 
& 

Organize 
Dau 



Review Dau 
with 

AsslsUnt Director 
of 

Dcfzupatlonal Ed. 



Graduate/ 
Completer 
Perfomanca 



Cost 
Factors 



Olv. Chair 



Div. Chair 
T/D Dean 



8th Week 
of 

Seeiester 



3th Meek 
of 

SeMster 



Deliver 
PrellBlnary Draft 
of 

Final Report 



Div. Chair 
T/D Dean 



lOth Week 
of 

Semster 



Submit Final 
Report to 
College V.P.I. 
& President 



Conduct 
Campus Program 
Review Heeting 



T/O Dean 



V.P.I. 



End of 
Sestester 



3nj Week of 
Following 
Semester 



Submit 
Synthesized 
Final Report to 
01 St. Director of 
Occupational Ed. 



T/D Dean 



4th Week of 
Following 
Senester 
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7q best copy available 



1. PROGRAM DEMAND 



In analysis of employment or manpower demand for products of 
a specific vocational education program, both present demand, 
as well as future (5 year) demand shall be considered- Demand 
trends in related occupations and cyclical economic conditions 
shall be reflected in the evaluation. 

It is suggested that enployment demand information be obtained and 
analyzed from the following sources, 

a. Texas State 1202 Commission Report for the Post-Secondary 
Educational Planning. 

b. Texas Employment Commission Data (Job Scene and Monthly 
Occupational Shortages*) 

c- Department of Labor Bureau of Labor Statistics- 

d. Local employer questionnaires. 

e. Newspaper classified ads. 

f. Advisory Committees comments. 

9- Tex'-SIS follow-up forms 
F05-1-E question 6 

F04-1.G question 1, 6, i 7 Section C 
h. Faculty and placement officer(s) comments, 
i- Other. 

The final report shall address the following questions. 

1- Is there sufficient demand to warrant the continuation of the program? 

a. Any expansion necessary to meet increased demand? 

b. Any trends that would signal the need for program modifications? 

2. Has demand decreased or are the future forecasts such that program 
reduction or elimination would be in order? 

a. Describe time phase down or out recommendations. 

b. Describe effects on personnel and or other programs. 
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PROGRAM DEMAND ANALYSIS SHEET 



Program_ 
College 



Date 



Texas 1202 Commission Report 



Texas Employee Commission 
Data 

Department of Labor Bureau 
of Labor Statistics 

Local Employer Questionnaires 

Newspaper Classified Ads 

Advisory Committees 

Tex-SIS Follow-up Forms 

Faculty 

Placement Officers 
Other 



VP 


p 


G 


E 




V^ 


P 


G 


E " 




































































































































































































1 



Overall demand summary 

Describe-any unique situations effecting demand (i.e* cyclical 
economic pfiase) . 

Future program recommendations as result of the demand analysis, 

* VP » Very Poor » Keenly Competitive 

P « Poor « Deminishing Demand 

G Good « Demand is present and stable 

E » Excellent = Demand is on the increase 
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Z. STUDENT KjTERgST 



Data for this component should be collected and analyzed from 
several sources. However, the following two basic sources a *e 
available through the assistance of district staff. 

a. Future student interest: Explorer Scout sponsored Career 
Interest Survey. {See survey form on page 7 and summary 
on page 8) 

b. Present and past interest enrollment records are on file in 
District and can be extracted and presented, showing a multi- 
ple years* historical pattern or trend. 

c. Other sources 

1. Interest in related Community Service courses. 

2. Interest in similar or related courses on other campuses and 
in other institutions. 

3. Has or is private enterprise responding to the need or interest 
In this program area by offering education or tra\ning? 

4. Have there been any requests from business ?»:id industry for 
specific training? 

5. Have college representatives called on businesses or industries 
for the purpose of extending the services of a program to them' 
(responses) 



6. What reactions, comments and/or results have staff received 
through recruiting efforts at area high schools? (Some information 
is available from the faculty self evaluation.) 

7. Has a Career Interest Survey been conducted soliciting student 
suggestions as to course offerings and preferred tim schedules? 



HISTORICAL DATA: 

Enrollment 

Certiflcatt levtl 
Foll-Tiiw 
Ptrt-TlBt 

Associate Level 
Full-Tine 
Ptrt-T1»e 

ToUl Head Count 

Awards 

Certificates 
Associate Degrees 
Total All Awards 



1972 1973 1974 









129 


117 


93 


142 


170 


209 


271 


287 


302 


22 


54 


70 


22 


54 


70 



1976 1977 









94 


56 


96 


199 


178 


198 


293 


236 


294 


70 


Is 


55 


70 


75 


55 
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3. INSTRUCTIONAL FACTORS 



(1) 
(2) 



This component will required the mc-t "on-campus" activity and it 
win also require the greatest amount of subjective data analysis 
The elements to be considered in this component are: 

a. Program and course goals and objectives shall be clearly written 
and agreed upon by college faculty, administration and advisory 
committees. Programs must show evidence that the goals and 
objectives are reflected in courses content and required 
performance of the students. 

™lf^ self-analysis and faculty analysis of program. (An 
example is shown on the following pages.) 

c. Curriculum analysis: 

A task analysis should be conducted if a recent task analysis 
ls not available. A copy of the task analysis shall be 
included in the evaluation materials. 

As a result of the task analysis, will curriculum revisions 
be developed and forwarded for approval? 

(3) The evaluation materials shall Include minutes of the Advisory 
Committee meeting in which the coimittee expressed positive 
reaction to and/or approved the curriculum. 

d. Present student evaluation summary of instruction (instructor, program. 

^' lfLl°.V°''y'^ -elected instruction related questions from the Tex-SIS 
?Sp L'^J follow-up forms are reconnended for analysis. 

(Some of these forms are not presently in use by the District.) 

2-6-0 Questions 2 i 7 
2-2-E Questions 1 i 5 

2- 1-E Questions 1 & 2 

3- 1-E Questions Sec. A 5, 6 i 7 
Sec. B 4 

3-2-C Questions Sec. A 4 

Sec. B 3 i 4 
Sec. C 2 i 4 
J-l-G Questions Sec. A 3 i 4 
Sec. B 5 
Sec. C 3 & 4 

,,,, Sec. 0 .1 4 2 

*(7) Tex-SIS F05-1-E Questions 3, 4. 5, 7 & 8 



(1) 


Tex 


-SIS 


(2) 


Tex 


-SIS 


(3) 


Je>t- 


-SIS 


(4) 


Tex- 


-SIS 


(5) 


Tex- 


■SIS 


*(6) 


Tex- 


•SIS 



♦included on the following pages. 
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4. GRADUATE/COMPLETER PERFORMANCE 



This component could be considered the most important factor In 
occupational program evaluation. College shall follow up on a 
significant percentage of their program graduates and or completers, 
a (45%) per cent return could be considered adequate for most 
programs. Colleges are CTicouraged to cooperate with and use the 
Tex-SIS follow up forms and systems, however when inadequate return 
data is not available colleges shall implement their own follow-up 
procedure using the Tex-SIS forms as a guide for questions. 

Follow up information shall be obtained from: 

a. Graduates 

b. Completers 

c. Employers 
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5. COST FACTORS 



In this component financial data shall be analyzed mainly through 
comparison procedures. Costs of the program shall be compared with: 

a. Other similar program costs on the campus. 

b. Other similar program costs with the District. 

c. Other similar program costs with the State. 

d. TEA Funding rates. (PrograniS can be considered "cost effective" 
If direct program costs are within approximately 752 of the TEA 
funding rate.) 

Unusually high and/or low cost effectiveness figures shall be explained 
and/or justified in the final report. 
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FINAL REPORT 



ERIC 



The final report shall be carefully developed from the data amassed in 
the Division Chairperson's office and will include the recomn^ndations 
of those persons attending the review meeting convened by the »Jce 
President of Instruction. The Division Chairperson, with the cooperation 
of the Tech/Occ Dean, shall Initiate the final report by organizing, 
analyzing, and synthesizing the data gathered. The report shall also 
inclide the Division Chairperson's and Tech/Occ Dean's recommendations 
and conmendations regarding the program. The report is then submitted 
to the Vice President of Instruction and the college President. The 
Tech/Occ Dean shall prepare a further synthesized version for submission 
to the District Director of Occupational Education. (A suggested guide 
for the final report is attached.) 

It is further suggested that the evaluation data be kept in an active 
file on campus fo? at least one year following the submission of the 
final report. The data will be useful in future program, budget, and 
personnel development activities. 
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FINAL REPORT 
(suggested guide) 
2 to 3 pages 



Name of program 
Campus 

Date{s) of evaluation 
Sunmary of program strengths 



Commendations: (List any particular person(s) or Incldent(s) In or 
surrounding the program that has added quality or enhanced the program 
beyond the norm.) 



Summarize any concerns Identified in the program. 



List corrective and/or program improvement action plans to overcome 
identified concerns associated with the success of the program. 



Discuss budget implications. 
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DALLAS COUNTY COMMUNITY COLLEGE DISTRICT 
OCCUPATIONAL PROGRAM PROFILE 



I. SAPABILITY 
A. INSTRUCTIONAL FACTCRS 
1. In what year did curriculum revisions occur in this program? 



2. What was revised and why was the revision necessary? 

3. Are there written objectives, stated in performance terms for both 
skill and knowledge levels, for every major course in the program 
on file with the office of VPI? 

B. INSTRUaiONAL PERSONNEL 

1. How many full-time faculty (PTE) taught 1n the program for: 

a. FaV« semester headcount and FTE for 81-82, 82-83 and 83-84. 

b. Spring semester headcount and FTE for 81-82, 82-83 and 83-84. 

c. Summer I headcount and FTE for 81-82, 82-83, and 83-84. 

d. Sunrier II headcount cr.d FTE for 81-82 , 82-83, and 83-84. 

2. How many part-time facul^ taught in the program for: 

a. Fall semester headcount and FTE for 81-82, 82-83 and 83-84. 

b. Spring semester headcount and FTE for 81-82, 82-83 and 83-84. 

c. Summer I headcount and FTE for 81-82, 82-83 and 83-84. 

d. Sunnier II headcount and FTE for 81-82, 02-83 and 83-84. 

3. In 1983-84, what percentage of contact hours <ere taught in yo'ir 
program by full-time instructors, full-time instructors on extra- 
service, and par^-tlme instructors? 

4. List the full-time faculty in the program involved In any profes- 
sipnal development activity related to their technical field and/ 
or instructional strategy (course, workshop, industry experience, 
etc.) to Improve or update their skills and/or knowledge during 
the past year? 

C. FACILITIES 

1. Quantify special facilities (excluding regular classrooms) which 
directly support tfiis program by providing the following informa- 



1981-82 



1982-83 



1983-84 



tion: 



1. 
2. 
3. 
4. 



Room Number 

Description of Facility 
Square Feet 
Other Quantifiers 
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2. Have you been unable to offer additiona"; courses/sections or turned 
away students due to specialized facility constraints this past Fall 
and/or Spring? If so, when? 

0. EQUIPMENT 

I. Is there a replacement schedule? 

Z. Have you been unable to offer additional courses/sections or turned 
away students due to specialized equipment constraints this past Fall 
and/or Spring? 

3. Oo you utilize specialized equipment off-campus to accommodate pro- 
gram demand? 

4. List approximate amounts of capital equipment (dollar amounts) pur- 
chased and percentage of accounts 2741 for the academic years 81-82, 
82-83 and 83-84. 



IL FUNDING 

A, STUDENT STATISTICS 

!• What were the enrollment statistics for the last three years (Fall, 
Spring. Summer I i II) (Source: STU 52600)? 

a. Contact Hours Total (Reoular, Co-op, Total) 

b. Headcount Unduplicated (Regular, Co-op, Total) 

c. Concurrent Non-credit (Headcount and Contact Hours) 

d. Related Continuing Education HEGIS Code 

e. Related Continuing Education (Headcount and Contact Hours) 

B. COST PER CONTACT HOUR 

!• Reimbursement rate for 81-d2, 82-83, 83-84. 

2. Cost per contact hour for 81-82, 82-83, 83-84. 

3. Median cost/contact hour (or average cost/contact hour) on statewide 
basis for 81-82, 82-83, 83-84. 

4. Contact hour projections (to wh^t extent has this program met contact 
hour projections) or (how do projected compare with actuals) for 
81-82, 82-83, 83-84. 



III. IMPORTANCE OF PROGRAM 

A. What type of activities have gone on within the past three years 
(1981-1984) which would demonstrate the college's coiroitment to this 
program? 

B. What is the composition of the program's advisory coimiittee? Please 
list area of occupation and ethnic background. 
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1. How many members attended (81-82 » 82-83, 83-84)? 

2. How many menders were new in each year (81-82, 82-83, 83-84 J? 

3. How many times did the advisory oxnmittee ^cr this program meet 
(81-82, 82-83, 83-84)? 



IV. JIARKH VIABILITY 

A. ' MARKETING 

1. How many special marketing efforts have been conducted by proc/ram 
faculty on direct behalf of this program within the past three years 
(High School visits. Business and Industry visits. Speeches, Adver- 
tisements, Other)? 

2. How often is the program brochure revised? 

3. How many high school seniors chose this area as a first career choice? 

B. OTHER 

L How many courses per year did this proqram conduct on another campus 
through home/host (81-82, 82-83, 83-84)? 

2. How many student job placements were made by: 

a. Program Faculty (81-82, 82-83, 83-84) 

b. Placement Office (81-82, 82-83, 83-84) 

c. Cooperative Education Office (81-82 , 82-83 , 83-84) 

3. How many students graduated from your program and (Source: Admissions 
Graduate Report) how many of your program graduates are employed in 

the occupation for which they were trained (81-82, 82-83, 83-84, Degrees, 
Certificates, Employed in Occupation) 

4* How much does a graduate earn (average per hour) who completes your 
program? (Source: Coop Director, Faculty and/or Advisory Conmittee)? 

5. How many students enrolled in courses in this program are currently 
employed in fields directly related to course of study? 

6. How many completers did the program have In 1983-8^? 

7. What special training programs for business and industry do you have? 
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DALLAS COUNTY COI<WUNITY COLLEGE DISTRICT 
OCCUPATIONAL PROGRAM ANAL.YSIS 



Strategic planning helps a college determine its long-range direction; 
define and decide between clear alternatives; and use available resources effec- 
tively. This means that program priorities need to be established and that 
resources may need to be reallocated. If this were not the case, it would mean 
that the status quo is presently suitable and will be suitable in the future. 
If the status quo was suitable, then the capabilities, funding, importance, and 
market viability of each program would be suitably aligned with the program's 
priority in the overall scheme of things. 

Program analysis is a vit'l step in the development of college plans. 
Program analysis should result in decisions about the priority of a program as 
well as an assessment of a program's capability, funding, importance, and market 
viability. While much of what happens in program analysis should be clearly 
rational, it is important that subjective judgements be rnade with objective 
information. In the DCCCD's strategic planning for occupational education, the 
program analysis component should use information^ collected by means of the 
Occupational Program Profile along with data from the College Service Area Fact- 
book and other available data sources. 
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The program analysis technique is a direct descendant of portfolio analysis 
techniques comnonly used in business. Variations of this approach have been 
used to good effect \r\ colleges and universities. One major Aspect of this 
approach is that it combines a market/political point-of-view with the rational/ 
scientific point-of-view so prevalent in most planning systems. The following 
sections describe a step-by-step process for conducting the program analysis: 
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Application of Criteria to Programs and Services, Prioritization of Programs, 
Cross-Ccv.iparison of Programs, and Reconmended Plans. 



APPLICATION OF SERVICE AREA CRITERIA 

The first step in the program analysis is to determine how a program relates 
to the service area through the consideration of student, general community, and 
business community criteria (ree Form A). The Occupational Education Task Force 
will use program information to reach consensus (or majority) in each of these 
areas. Th3 applicat'^on of these criteria, as described in detail below, will 
result in the assigmient of service area priority to each program at a college. 

A. Criteria 

Stydent. All occupational pro^ sms should provide current and up-to-date 
skills and knowledge. Additionally, the programs should provide training in 
occupations where job demand is sufficiently high to assure good "cyment 
opportunities and wages. 

General Community ^ Occupational programs should be meeting needs identi- 
fied in the cnmnjtnity which the college serves. Consideration should be given 
to whether the DCCCD/College is the only institution that can or is providing 
the program, and whether it is iriportant to maintain or ignore a program. 

Bu siness Community . Occupational program offerings should tie in directly 

with the needs^of the bus-^ness community. Each College should assess its ser- 
vice a^ea and determine what its priority is regarding (a) expansion of services 

to business and industry groups, and (b) increased involvement with business and 
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industry. Each College should also determine as appropriate, its involvement 
in curricular offerings related to high technology and entrepreneurship. 

B. Process 

•The OETF should invite the Division Chair and program faculty to partici- 
pate in the analysis. The following steps should be used: 

1. Rate the program in Column 1 using a scale of 0-5 with 5 being the 
highest. 

2. Multiply the rating by the weighted factor (x) to determine the score. 

3. Follow this procedure through the first five columns. 

4. In order to proceed, you must assign a weight factor to eac^i column to 
a maximum of eight points. 

5. Follow the same steps as before— multiplying the criteria rating by 
the weighted factor and writing in the score— for each column. 

6. In the last column, total all the scores from each column. 

PRIORITIZATION OF PROGRAMS 

In order to prioritize all the programs at your college you will list 
them from the highest to lowest score on Form B. Group the programs into three 
categories (High, Medium, or Low Priority) by placing a check under the appro- 
priate column. Thus, the OETF will determine what priority a program has for 
the college service area . That is, from the point-of-v1ew of individuals and 
groups being ser/ed (or not served) what is the priority of a certain program 
or service? The service area priority may be (and often is) different from 
the importance of a program or service within the college. 
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CROSS-COHPARTSON OF PROGRAMS 



The cross-comparison process uses a series of matrices to ccaipare service 
area priority (external focus) to several Internal factors (capability, funding, 
importance/central ity, and market viability). This assures that the market/poli- 
tical factors contain ' ' in the service area priority are given continual Impor- 
tance. 

The OETF comes to consensus on the relative capability, funding, importance/ 
central ity, and market viability of each program by giving each program a High, 
Medium, or Low rating on each of those Internal factors by placing a check-mark 
on Form C in the aporopriate cell of each matrix, i.e. High Priority/High Impor- 
tance, and High Priority/High Market Viability, This is, of course, easily 
said but not so easily done. Once this has been accomplished the OETF can get 
down to the whole point of the program analysis, making recommendations about 
the allocation of resources to the programs analysed. Definitions of the inter- 
nal factors are listed below. 

Capability . Based on data from the Profile (and other sources) along with 
the combined wisdom of the Task Force members, "capability" is the ability of a 
program or service to do the job it ought to be doing. The fact that the capa- 
bility of a program may be judged to be low should be seen simply as a lack of 
resources for the program. This is an assessment of relative strength and an 
indicator of the extent to which the college hat devoted quality management, 
staff depth, instructional materials and equipment, and so forth. 
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Fiindina. This is the perception of the Task Force as to the relative 
strength of funding for a program or service. As compared to other programs 
and services within the college, how well funded is this one? All programs 
need more funding, of course, but some need it more than others. 

Importance/ Central ity . Some programs and services are simply important to 
the college despite low volume and an uncertain future. Other programs and ser- 
vices are important because of obvious factors such as increasing enrollments, 
high job demand, and so forth. The Importance/central ity factor makes explicit 
the central ity of a program to the college . For example, the University of 
Houston has recognized the importance of the foreign language curriculum despite 
low student interest, enrollment, and job demand simply because a university 
must offer foregin language instruction. There may be programs or services with- 
in colleges in the DCCCD which are highly important for similar reasons. 

Market Viability. Market viability is the perception of the Task Force as 
to the relative stength of a program or service to survive a.id grow in response 
to demand (not need) from the market. This factor should be assessed based on 
Task Force perceptions of trends In student interest, job demand, salaries, and 
so forth. 

RECOMMENDED PLANS 

Looking at the cross-comparisons made for each program, the GETF makes 
specific recormendaticns about improving, maintaining, or relaxing capability, 
funding, importance/central ity, and/or market viability within each program 
and translates those recommendations into suggested activity plans. At this 
point the analysis is over and administrative decision-making, planning, and 
implementation begin. 
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College: 



FOAM A 

APPIICATIOH OF CRITERIA 
YO PROGRAMS AND StRVIt.S 

CHtsHa (0-S, with 5»Hfgh€5t) 



BEST COPY AVAILABLE 



STUDENT 



PROGRAM/SERVICE 



Pro^fM (4) 
Provides 
Current 
Knowledyt 



Ct. Scort 



(5) 



Job 
DeMnd 



Ct. Score 



GENERAL COHHUNfTY 



Progran Meets (4) 
Perceived 
Tr«lRln9 Needs 
of CoonunUy 



Ct. 



Scort 



To Uh«^^ (2) 
Extent Is the 
DCCCO the 
Only Provider 



Ct. 



Score 



Severe Negative (2: 
Reection Will 
FolloM If Abandoned 
or Ignored 



Ct. 



Scort 



(Max. 8) BUSINESS COMHIMITY 



Instruc- ( ) 
tional Ser- 
vices to 
Bus/Ind. 

Ct. Scors 



( ) 

Involvenent 
Ml 6u$/lnd 

Ct. Score 



( ; 

Hi-Tech 
Eaphasis 

Ct. Score 



( ) 



Entreprt* 
neurship 



Total 

Score 



Ct. 



Score 



00 



87 




Ct. (Criteria) x lx) (Weight) • Score 



88 
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FORM B 

College: 

RANKED INVENTORY OF PROGRAMS AND SERVICES 



1 
2 
3 
4 
5 
6 
7 
8 
9 
10 
11 
12 
13 



PROGRAM/SERVICES IN SERVICE AREA 

DESCENDING ORDER PRIORITIES 



PROGRAM/SERVICES SCORE HIGH Ymm 
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lege: 



Program: 



mm 



sitvici ASIA nimiTi 



SUYtCI AREA ffitOim 



HIGH 



>S0 



l(M 



NIOt MIOIW IM 



»€0 



^ mo 



i 
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A 3-TIERED PROGRAM REVIEW SYSTEM 
TRITON COLLEGE 



Illinois CoBmunity College Mandate 

Beginning FY 1984, the Illinois Community College Board 
mandated that all community colleges conduct formal program 
review of at least 20 percent of their programs and academic 
support areas each year. 



Coiaponent s of Pro gram Review 

Three areas were emphasized by the ICCB in terms of expected 
areas of review: 



(1) Student and Employer Demand for Program 

(2) Program Cost 

(3) Program Quality 



In Illinois, the five primary reasons for statewide review 



are: 



(1) Assist in Statewide Planning 

(2) Justify State/Federal Expenditures 

(3) Establish Priorities 

(4) Assure Efficient Program Operation 

(5) Assure Local/Statewide Attainment of Educational and 
Employment Needs 
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Choices 



Triton College considered two systems for responding to the 
required program review task: COMPREHENSIVE REVIEW OF EACH 
PROGRAM AND SUPPORT AREA EACH FIVE YEARS; 3-TIERED PROGRAM REVIEW 
SYSTEM (Also considered "comprehensive" but multi-level). 



CHOICES 



COMPREHENSIVE 
(FULLY) 

Yearly Activity (Constant) 

20% (Programs/ 

Disciplines) 15 

20% (Support Areas) . . 6-7 

Total 21-22 



COMPREHENSIVE 

(3-TIERED) 

SCREENING/FOCUS/ 
COMPREHENSIVE 

Yearly Activity (Varies) 

100% (Programs/ 

Disciplines) 75* 

'20% (Support Areas) . . 6-7 

Total 81-82 



*Comprehensive .... 6 
*Focus 53 



Thus, selection of the fully comprehensive system would have 
meant that 15 programs/disciplines + 6-7 support areas would have 
comprehensive review each year. 

The 3-tiered approach resulted in 75 programs/disciplines 
having Tier I (screening); 53 having Tier II (focus on 1 or more 
of 11 areas); and 6 having Tier III (comprehensive — total 
review) . 
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p;%tj Qnale for Selection of 3-Tiered Syst em; 



An analysis of the strengths and weaknesses of each system 
resulted in the conclusion that the 3-tiered system was a more 
effective and efficient one. 



SYSTEM ANALYSIS 



Comprehensive (Fully) 



Comprehensive (3- tie red) 



+1. 100% comprehensive 
each five years (20% 
each year) 

+2. Each program evaluated 
in a multi-faceted 
manner 

-3. Weak programs slip by — ^ 
would be five years 
before next review 

-4. Strong programs demand 
same staff time as weak 
ones 

-5* Trends difficult to 

ascertain with only one- 
fifth reviewed yearly 

-6. Needed attention to weak 
areas diluted by massive 
paper ( reading/analysis) 

-7. Process tends to become 
mechanical 



+1. 100% screened yearly — 
"snapshot" of each 
program developed 

+2 . Management-by-exception 
approach, "Don't fix it 
if it's not broken" 

+3. Weak get attention yearly 



+4. More productive use of 
staff realized 



+5. Signals college-wide 

areas needing attention 



+6. Trends evident because 
basic data collected 
yearly on all programs 

+7 . Everyone focuses 
attention on 
program/discipline 
strengths/concerns on a 
yearly basis 

-8. Multi-faceted, in-depth 
analysis of each program 
may not occur 
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roip ponents of 3-Tiered System; 



The screening model is based on a PIMA COLLEGE, ARIZONA, 
model and provides computer-generated information, typically on a 
four-year trend basis, on the following areas: Number of majors 
in program, enrollments in courses in specific program or 
discipline, class size trend, cost of program, ratio of full-time 
to part-time faculty, turnover of part-time faculty, numbers of 
students successfully completing courses in program/discipline, 
number of graduates, rate of completion of program, job 
placement, and use of instructional space. 

Following generation of the screening model data, a two-page 
analysis (strengths/concerns/conclusions) is prepared for each 
program/discipline by the Office of Staff and Program 
Development. This analysis, requiring about one full week, along 
with a one-page computer-generated sheet on student 
characteristics of the program—median age, load taken, sex, 
racial breakdown, residence (in or out of district), time of day 
classes held, educational intent of students, and employment 
status of students— is sent to appropriate department 
chairpersons. 

The chairpersons, along with members of their department, 
prepare two items: id) A response to the two-page analysis 
Tfreauentlv there is' information available that explains some of 
the screening model coYicerns) ; (2) A one-page analysis of the 
strengths and weaknesses of the program/discipline as seen by the 
department and covering items that may not have been addressed in 
the screening model. This second report is particularly 
important in identifying program quality factors missed by the 
heavily quantitative screening model. 

Following receipt of departmental response, a decision is 
made on the action to be taken on each program/discipline, 
possibilities include: 

(1) Repeat the screening model only (automatically done for 
all programs) (Tier I) . 

(2) conduct a focus study (Tier II) of one or more of the 11 
areas included in the screening model. Past experience 
indicates that most programs would need focus studies on 
no more than 2 or 3 of the 11 areas. 

(3) conduct a comprehensive review (Tier III) of the entire 
program. (This includes a major study and includes 
components commonly found in most program review 
processes . ) 
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pxpanded Screening Model (Tier I) ; 



Because the screening model tends to emphasiz . limited 
amount of quantitative information, it became evidenc that 
additional information was needed, both of a quantitcitive and 
qualitative nature, to begin to address the more difficult 
"quality" issues. 

As a result, Triton College has now developed an expanded 
screening model, which will continue to be expanded as additional 
data is gathered. Listed below is an overview of this change: 



EXPANDED SCREENING MODEL 



Basic; 

*****Screening Model Data on 11 Areas 
*****Two-page Analysis of Areas of Strength/Concern 
*****Response of Program Personnel to Analysis 



Expanded; 

*****One--Page Statement of Strengths/Concerns as 
Perceived by Program/Discipline 

*****One-Page Computer-Generated Student Profile Sheet 

(1) Age, Sex, Race, Part-Time/Full-Time Status 

(2) Employment Status, Hours Employed, 
Educational Intent, etc. 

*****One-Page Analysis of Employment Status of Latest 
Follow-Up Study on Occupational Graduates 

*****Three-year Equipment Replacement Plan 

*****Program/Course Objectives 
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support Areas: 

Because of the diversity of the 33 academic support areas at 
Triton (e.g.. Learning Resource Center, Learning Assistance 
Center, Career Planning & Placement, etc.), special review 
instruments need to be used for each of these areas, which dfi 
proceed through a regular five-year review cycle. 



Model; 

The following page contains a graphic model of the 3-tiered 
process indicating how the 75 program/disciplines and 33 support 
areas at Triton are reviewed. 
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PROCESS 



Screening Model Osed 

for All Academic 
Progrzuas/Disciplines 



Programs/ 
Disciplines 



Tier I 
(Yearly) 



Tier II . 
(As Needed), 




Tier III 
(As Needed) 



Repeat 
Screening 
Model 




Implement 
(Comprehensive) 
Phase II 



Expanded 
Screening 

Model 

Data 



Academic Support Areas 
Co«prehensive 
Evaluation 



-© 



33 ) Areas 



Learning Resource Center 



Financial Aids 



Employee Developm ent Institute 



rounselinq 



Etc. 
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<?^^ffffia ry of 1984 Evaluative Activity Reaultinq froa Process s 



As indicated below, only 6 prograuns were identified as 
requiring compirehensive review, compared with 2 5 had Triton 
elected the five-year, 20 percent process* Even more important, 
although 53 programs required focus studies, 48 of these required 
attention to 3 or less of the 11 areas. The amount of time 
required for a focus study is typically minimal. . 



COMPREHENSIVE 



Accounting 
Data Processing 

Industrial Supervision 6 Management 
Engineering Design/Drafting 
Welding 

Recreation & Leisure 



FOCDS 



One Evaluative Area 


21 


Two Evaluative Areas 


11 


Three Evaluative Areas 


16 


Four Evaluative Areas 


4 


Five Evaluative Areas 


_i 




53 



NEITHER 

15 
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Y^ar-to-Y ear Analysis: 



The first year the 3-tiered system was in place, a three- 
level rating system was used, with: 



Strong 

Neutral N 

Weak • . . . - 

It became apparent, when looking at the data for the current 
year, that a strong area may still look strong but may have 
slipped considerably. Similarly, a weak area may scill be weak, 
but may have improved. 

A more complete system was devised which now uses the 
following scale: 



Strong area S+ 

Strong but less than last year S 

Strong but significantly less than last year . . S- 

Drop from strong to neutral SN 

Drop from strong to weak . . . SW 

Increase from neutral to strong NS 

About the same , . . . . N 

Drop from neutral to weak NW 

Weak but better than last year W+ 

Weak and becoming weaker W- 

Increase from weak to neutral WN 

Increase from weak to strong WS 

Not applicable NA 



It is now possible for a program director to move through 
each of the 11 areas and know exactly what has occurred from last 
year to this year. The attached summary places all programs in 
Nursing, Allied Health, Personal & Public Service on one page for 
rapid area analysis. 

The second attachment also is valuable in providing actual 
figures for comparative purposes. 
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SCREENING MODEL INFORflA- 
tion. YEAR 
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' [I '- 1 1 — - — rt — YoarV S- (Strong, Significant! 

"^c;. fst-ono Area); S (Strong Area, Less ^^^f .^^f.; '^^^ ! sw (Drcn from Strong to 

s ' hrLasfkar) SN .(^-P.^rfs "X^llJ: ^rn-; " (Drop from f:eutr- 

Weak); NS (Increase fro^- Meutral to f^^f^J. ,^ y „. (vieak % . 

"trSek); US (increase fro. Ueak to 

Becoming Weaker); UH Hncrease/rom^We.k^t^^^^ M/MlABlt 
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BEST COPY AVAILABLE 



I o 




PROGRAM REViEH REPORTS/ SAMPLES 



The following pages contain a variety of materials related 
to the 3-tiered program review process including: 

I. Triton College Proogaa/Discipline De partnent 
Screeni ng Model 

Introduction 

Method for Deriving Scores on 11 Areas 

Program Screening Worksheet {Sample: Child Care) 

Program Screening Model {Sample: Astronomy) 

Curriculum Profile Child Care^ Pall, 1984 

Latest Graduate Follow-up Report — Child A.A.S. and 

Child Care Certificate Graduates 
Two-page Summary Prepared by Office of Staff & Program 

Development Based on Child Care Screening Data 

Not included ; 1-2 page Departmental Analysis of 
Strengths and Concerns not covered in Screening Model 



II. Pocas Sn ^wary {Characteristics noted by times 
indicated) 



III. Comprehen sive Program Review 



98 



ERIC 



103 



TRITON COLLEGE 
PR0GRAJ1/DISCIPLINE/DEPARTMENT SCREENING MODEL 



Tntrcduction 

Program evaluation has become a vital, and yet enormously 
time consuming* activity for most institutions of higher education 
in the United States. Like many other institutions, Triton 
College found itself with a program review process so unwieldy as 
to be almost unmanageable. 

Two major steps were taken to remedy this problem. The "in- 
depth" program review process has been modified and greatly 
streamlined. In addition, a quick, "screening model" has been 
developed which will allow us to take a preliminary "snapshot" 
look at all programs before going into depth for any one program. 

The Prograro/Discipline/Department Screening Model has been 
adapted from a model used at Pima College in Arizona ("A 
Screening Model for Community College Program Evaluation," 
Schultz and Webb, New Direc tions for Community Colleges, 25, 
1S79). To quote from that article: 

[. . . what is needed] is a procedure that permits a 
systematic general review of thei^ programs to identify 
those which appear to need in-c'.^pth investigation. 
This process of "flagging" penaits an institution to 
concentrate its time and resources on a manageably 
number of programs for detailed evaluation. In turn, 
it assists the managers in making informed decisions as 
to which programs can continue basically unchanged and 
which ones should be strengthened, modified, or 
eliminated. 

In order for this general review process to be 
functional, it needs to meet at least two conditions: 
it must (1) have sufficiently broad parameters so that 
it can be applied to a wide range of programs, and 
(2) it must use a data base that already exists for the 
institution, or can be readily developed. 

The screening model is based on five components of program/ 
discipline/department activity: 

1. Enrollment Characteristics 

2. Cost 

3. Instructional Conditions 

4. Student Success 

5. Utilization of Instructional Space 
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On the pages that folloWf the components of the model (there 
is both a School of Arts and Sciences and a Career version) are 
described/ and samples of both versions are presented. The basic 
intent is to use the model in the summer of each year/ after the 
relative cost report (which supplies much of the data) is 
available. 

Some caveats are in order. 



1. The screening model provides a starting point in the 
evaluation process and is intended as a guide to further 
study/ not as a study in itself. 

2. The model is not a normative instrument. That iSf it is 
not intended to prescribe what each prograun "should" 
score. 

3. It is impossible to score 100 percent of possible 
points. There are unavoidable tradeoffs built into the 
model (e.g.f between full-time faculty utilization and 
cost) . 

4. The model works best as a diagnostic tool for a given 
program. It is not essentially a comparative :^ 
instrument. In particular/ comparisons of Career ' 
programs to SAS programs/disciplines/departments shduld 
not be made/ as the model is quite different for the two 
schools. 
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PROGRAM/DISCIPLINE/DEPARTMENT SCREENING MODEL 
CAREER EDUCATION 

for Deriy lm SC^rffP Areas 

1. rijrriculum Trend 

Takes curriculum enrollment for the most recent fall terra and 
divides that by the average curriculum enrollment for the three 
previous fall terras. 

2. Course Trend 

Takes program course enrollments for the raost recent fall term 
and divides that by the average program course enrollments for 
the three previous fall terms. 

3. ri ass Size Tre nd 

Takes the average program class size for the most recent fall 
term and divides that by the average program class size for the 
three previous fail terms. 

4. C9 Pparati ve Cost 

Divides the program's relative contribution per credit hour by 
Career Education's relative contribution per credit hour. 
Relative contribution does not attempt to accoun-: for fixed 
costs. 

5. p^in -Time F?^g""ttY Utilization 

Takes the program's ratio of sections taught by full-time faculty 
to sections' taught by all faculty and divides it by Career 
Education's ratio of sections taught by full-time faculty to 
sections taught by i>ll faculty. 

6. r^n tinnlnq Part-Time Facultv 

Takes the program's ratio of total part-time faculty to new part- 
time faculty and divides it by the Career Education's ratio of 
total part-time faculty to new part-time faculty. 

7. Cnui ^se Co mpletion Rate 

Divides passing grades (A, B, C, D, P, ADD) in the program for 
the raost recent fiscal year by total grades issued m the 
program. 

8. rnpipiPf ion Trend 

Takes the nuraber of prograra graduates for the most recent 
academic year and divides by the average number of program 
O graduates for the three previous academic years. 

10106 



9, Completion Rate 



Takes the number of program graduates for the most recent 
academic year and divides it by the number of program majors in 
the fall termr three years prior. This ratio is then divided by 
the average of the ratios for the three previous years* 

10. Graduate Success 

Measured by looking at the proportion of graduates employed in a 
field related to their progreun of study. Divides the proportion 
for the most recent follow-up by the average of the proportions 
for the three previous follow-ups. 

11. Space Dti lization 

Divides the seats taken m the program for the most recent fiscal 
year by the total seat capacity for courses in the program. 
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TRITON COLLEGE 



PHASE II 

COMPREHENSIVE PROGRAM BVALDATION 



pro gram D escription 

A. Prograin Components {Degrees, Certificates, Options, 
Date Established, Total Credits in Program, Outside 
Program) 

B. Primary Program Employment Gcal(s) 

C. Program Admissions/Retention Requirements 

D. Courses in Program 

1. Number approved 

2. Number offered per term (range) 

3. Number serving primarily as service courses for 
other programs 

L. Relation of Program to External Constituencies 

1. Accrediting agency (date/last review) 

2. Senior college transfer articulation, if any 

a. Program articulation 

b. Course articulation 

3. Advisory committee data 

4. Community resources used 

5. Contracts/agreements with other agencies 

F. Additional Comments 
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Phase II 

Comprehensive Program Evaluation 



II. Prooraa Need 

A. Program Enrollment (Fiv6-Year Historic & Three-Year 
Projection) 

B. ^- Number of Completers (Five-Year Historic & Three- Year 

Projection) 

C. Number of Programs (Five-Year Historic & Three-Year 
Projection) 

D. Summary of Student Success in Program's Courses (Last 
Three Years) 

E. Employer Demand 

1. Local district openings (current & five-year 
outlook) 

2. Regional openings (current & five-year outlook) 

3. State openings (curreni: & five-year outlook) 

F. Source of Demand Statistics 

G. Enrollment by Specific Course (Five-Year Historic & 
Three-Year Projection) 



III. Pr ogram Costs 

A. Unit Cost/Credit Hour (Five-Year Historic & Three-Year 
Projection) 

B. Unit Cost/Completer (Five-Year Historic & Three-Year 
Projection) 

C. Number of Faculty (Five-Year Historic & Three-Year 
Projection) (FT, PT, FTE) 

D. Class Size (Five-Year Historic i Three-Year Projection) 

E. FTE Faculty/Student Ratio (Five-Year Historic & Three- 
Year Projection) 

F. Additional Comments 
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Phase II 

Comprehensive Program Evaluation 



IV. Prograa Quality 

A. Full-Time Faculty- Degraes 

" Years Work Experience 
■ Years Teaching 
Years at Triton 
' * Professional Development Activities 

B. Adjunct Faculty (same data as full-time) 

C. Student Ratings (13 item scale)— FT Faculty 

» • m m • __p.p Faculty 

D. Student Rating of Instructional Materials, Physical 
Conditions in Classroom/Lab 

E. Overall Equipment Evaluation 

F. List of Critical Equipment Needs 

G. Assessment of Facilities 

3. Assessment of Clerical & Other Support Staff 

I. Student Preparation 

1. Ratio of program completers/FTE program enrollment 
(five-year historic) 

2. Number of completers employed (five-year historic) 

3. Number of completers continuing education (five- 
year trend) 

4. Mean salary of completers employed full-time in 
field 

J. Program Rating by Alumni, Current Students, 

Business/ Industry, Accrediting Agency, Noncompleters 

K. Additional Comments 



V. Program Strengths (One-Page Analysis) 



VI. Program Weaknesses (One-Page Analysis) 



VII. ?hree-Year Program Plan (Two Pages) 



SELP-STDDY 



INSRDCTIONAL PROGRAMS: ARTS AND SCIENCES DISCIPLINES r CAREER 
EDOCATION SUPPORT DISCIPLIHESr DEVELOPMENTAL EDOCATION, 
CONTINUING EDUCATION 

Name of Prograun:_ 

Name of Progrjun Head: 



I. Proqraa Description : 

A. Prograip. Components 

Title of Degrees Date Credits Required 

and/oc Cgctjfjcatgs SStablishe^ for completion 



OR: Program Scope/Areas of Study 



B. Primary Program Goal : 



C. Admission s/Selection Criteria and Retention 
Requirements : 



D. 



Courses Offered: Number Num 


ber Offered franael 


Approved 

1. As general educa- 
tion requirements 
for transfer decree 


Fl 


[Wtr) 


Spr 


Smr 

• 


Other 


2. As service courses 
for degrees/ 
certificates 












3. As General studies 












4. As remedial 












5. As ABE/ASE 












6. As ESL 













ERIC 



106 



111 



Relations with External Constituencies: 



1. Senior college transfer articulation: 

(Baccalaureate programs should list courses not 
approved for transfer.) 



2. Contracts/agreements with other 
agencies/ institutions : 



3. Student use of community resources (practicums, 
field trips, speakers, libraries, etc.): 



Additional Comments : 



107 

ii2 



II. proqrMt Need 



Historic Data 



Projections 



ERIC 



A. 



C. 



D. 



Summary of Student 
Success in Courses 
in the Program 
(Past Three Years) 



19— 


IS- 


IS— 


IS- 


IS— 


IS- 


IS— 


IS— 


Program Enrollment 
(where applicable) 

1, # or ruii-time 
students 

















2. # of part-time 

students — _ 
















3- PTE students 
















Completers 

1. # of degree 

recipients 
















2. #01: certificate 
completers 
















3. # of course 
completers by 
discipline 
















# of credit hours 
(annual) (program 
or disclDlineJ . 












• 





Grade 


IS— 


IS— 


is_ 


% A 








% B 








% C 








% D 








% P 








% F 








% W 








% I 








% R 
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I % 



Historic Data 



Projections 



19— 



E. Enrollment in 

Courses Grouped by 
Discipline: 



19. 



19_ 



19. 



19_ 



19_ 



19. 



19. 



F. Additional Comments: 



III. pirogram Costs 



Historic Data 



Projections 



19. 



19. 



19_ 



19_ 



19_ 



19_ 



19. 



19. 



A. Onit Cost per 
Credit (Contact) 
Hour (Program and/ 
or Discipline) 
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Historic Data Projections 



19_ 


19-^ 


19_ 


19-^ 


19--^ 


19^ 


19^ 


19>-. 


Unit Cost Per 
Program Completer 
(Wlieta Appropriate) 
















Number of Faculty/ 
Professionals 

1 • Full-time 
2- Part— time 
3. FTE 
















Number of Support 
Personnel 

1 • Full-time 

2. Part— time 

3. FTE 
















Class Size (by 
Discipline) 

1 • Mean 

2. Hioh 

3. LOW 
















1 • Mean 
2- Hiah 
3. Low 














— 


1 • Mean 

2. High 

3. LOW 
















1 • Mean 

2. High 

3. Low 
















1 • Mean 

2. High 

3. Low 
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Historic Data 



Projections 



19- 



P. PTE Paculty/Staff 
to Student Ratio: 



19. 



19_ 



19. 



19__ 



19. 



19. 



19. 



G, Additional Comments: 



IV. 
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Program Quality 
A. Faculty and Support Staff 
1. Education 



Less than 
Bachelor's 


Bachelor's 


Master's 


Master's 
Plus 


Doctorate 


Faculty (full-tiine)' 
Faculty (adjunct) 
Mid-Management ^ 
Classified 
Hourly 











2. Years of Teaching 
Experience 



Faculty (full-time) 
Faculty (adjunct) 
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Mean 


High 


Low 




1 







< 



3, Professional Development 



Professional Staff {Full-Time) 
In-s^orvioe training 
Professional Associations/ 
Societies 

Professional Staff (Part-Time) 
In-servioe training 
ProfessicMial Associations/ 
Societies 

Mid-Managenentr Classified Staff 
In-service training 
Professional AssociaticMis/ 
Societies 

Hourly Staff 

In-service training 
Professionad Associations/ 
Societies 



Number Participating Annuailly 



Mean 



High 



Low 



(Section B below is intended for the Continuing Educaticn program only.) 

B. Faculty Assigned to Occupational Courses 
1. Education 



ERLC 



Less than 
Bachelor's 


Bachelor's 


Master's 


Master's 
Plus 


Doctorate 













2. Years of Experience 



a. Working in field 
(past saven years) 

b. Teaching in field 

c. Teaching at Triton 
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Mean 


High 


Low 
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student Ratings 
of Instruction 
(Full-Time Faculty) 



Always 



Item #1 
Item #2 
Iten> #3 
Item #4 
Item #5 
Item #6 
Item #7 
Item #8 
Item #9 
Item #10 
Item #11 
Item #12 
Item #13 

Overall Rating 
of Instructors 



Usually 



Percent Rattv^ 



Sometimes 



Never 



Doesn't Apply 
or Don't Know 



Very 
Effective 


Good 
Instructor 


Average 


Below 
Average 
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student Ratings 
of Instruction 
(Part-Tine Faculty) 



Alweys 



Item #1 
Item #2 
Item t3 
Item *4 
Item #5 
Item #6 
Item #7 
Item *8 
Item #9 
Item #10 
Item #11 
Item #12 
Item #13 

Overall Rating 
of Instructors 



Usually 



Sometiioes 



Never 



Doe£sn't Apply 
or Don't Know 



Rating of 

Instructional 

teterial 



Textbodcs 
Assigned readings 
Transparencies 
Handouts 

FiM, filmstripSf 

slides 
Lab equipnent 



Vory 
Effective 


Good 
Instructor 


Average 


Belov 
Average 










Excellent 


Good 


Fair 


Poor 


Not Osed 
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Rating of Physical 
Condition in 
Classroom Lab 



Excellent 



Cleanliness 



Good 



Fair 



Poor 



Not Used 



Ttecperature 



Lighting 

Overall 
Attractiveness 

Provide an overall evaluation of equipment used in the program* Include 
any critical equipnent needs: 



Facilities 







Adequate 


Inadequate 


Conments 


1- 


ClassrooRS 










Laboratories 








3. 


Offices 









Clerical and Other Support Staff 
Number: 



Adequacy of Number: 



Overall Quality: 



Adequate 


Inadequate 


1 — 

Comnejits 








Good 


Fair 


Poor 


C^-^iments 
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G. Student Success 



Historic Data 



19 



19. 



19_ 



19. 



19_ 



1. Ratio of conpleters 
to airollinent 
(annual headcount) 

2. # of coTpleters: 
(respond as 
applicable) 

Employed full-time 

Bmployed part-tiine 

CcHitinuing Education 
(transferring) 

None of above 

H. Additional Consents: 



V. p^QC fram S trengths 

In no more than one page, describe the strengths of the 
program. 

VI. Program weaknesses 

In no more than one page, describe the program's weaknesses 
or the areas of the progra^m which need to be improved. 



VII- Three-Ye ^r pfograa Plan 

In no more than two pages, describe the plans for action 
to be taken to remove program weaknesses and make improve- 
ments in the program. (Include program scope and require- 
ments, instructional methods, and resource requirements.) 
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Appendix D: 

Member Institutions of the 
National Postsecondry Alliance 
(1985-86) 
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Membership List for 1985-86 



1- Alamo Community College District, San Antonio, Texas 

2- Anchorage Community College, Anchorage, Alaska 

3- Atlantic Community College, Mays Landing, New Jersey 
4. Bergen Community College, Paramus, New Jersey 

5* Bessemer State Technical College, Bessemer, Alabama 

6- Biq Bend Community College, Moses Lake', Washington 

1. Boise State University, School of Vocational-Technical 

Education, Boise, Idaho 
8. Brevard Community College, Cocoa, Florida 

^. Catawba Valley Technical College, Hickory, North Carolina 

10. Catonsville Community College, Catonsville, Maryland 

11. Central Arizona College, Coolidge, Arizona 
12- Champlain College, Burlington, Vermont 

13. City Colleges of Chicago, Chicago, Illinois 

14. Clark Technical College, Springfield, Ohio 

15. Columbus Technical Institute, Columbus, Ohio 

16. Cuyahoga Community College District, Cleveland, Ohio 

17. Dallas County Community College District, Dallas, Texas 

18. Del Mar College, Corpus Christi, Texas 

19. Durham Technical Institute, Durham, North Carolina 

20. Eastern Iowa Community College District, Davenport, Iowa 

21. Florida Junior College at Jacksonville, Jacksonville, Florida 

22. Greenville Technical College, Greenville, South Carolina 

23. Guilford Technical Community College, Jamestown, 

North Carolina 

24. Hocking Technical College, Nelsonville, Ohio 

25. Jefferson Technical College, Steubenvil le, Ohio 

26. Lakeland Community College, Mentor, Ohio 

27. Lewis-Clark State College, Lewiston, Idaho 

28. Maricopa Technical Community College, Phoenix, Arizona 

29. Mercer County Community College, Trenton, New Jersey 

30. Mississippi Gulf Coast Junior College, Perkinston, 

Missiirsippi 

31. Orangeburg-Calhoun Technical College, Orangeburg, 

South Carolina 

32. Owens Technical College, Toledo, Ohio 

33. Patrick Flenry Community College, Martinsville, Virginia 

34. Community College of Rhode Island, Warwick, Rhode Island 

35. St. Louis Community College, St. Louis, Missouri 

36. South Puget sound Community College, Olympia, v;ashington 
3 7. Community Colleges of Spokane, Spokane, Washington 

38. Tarrant County Junior College, Fort Worth, Texas 

39. Texas state Technical Institute-Sweetwater , Sweetwater, Texas 

40. Triton College, River Grove, Illinois 

41. fitah Technical College at Provo, Provo, Utah 

42. Walla Walla Community College, Walla Walla, Washington 
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